Re: Review Request 49676: Add atlas-application config sections to all services that run Atlas hook, e.g., Hive, Falcon, Storm, Sqoop

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49676/
---

(Updated July 6, 2016, 8:29 p.m.)


Review request for Ambari, Madhan Neethiraj, Robert Levas, Sumit Mohanty, 
Swapan Shridhar, and Suma Shivaprasad.


Changes
---

Added addendum patch


Bugs: AMBARI-17573
https://issues.apache.org/jira/browse/AMBARI-17573


Repository: ambari


Description
---

Currently, Atlas hooks that run in Hive, Falconm Storm, Sqoop processes 
reference atlas-application.properties file from Atlas server config location - 
/etc/atlas/conf/atlas-application.properties.
Not all properties in /etc/atlas/conf/atlas-application.properties are required 
in hooks and some of these properties are sensitive enough not to expose them 
to hooks/clients.

To address this concern:
1. atlas-application.properties should be added as a config section in each of 
the host component's that run Atlas hook - Hive, Storm, Falcon, Sqoop
2. These new config sections will only include properties that are required to 
the respective hooks
3. During initial deployment, Ambari will initialize these properties with 
values in Atlas server configuration.
For each one of those services, create a config type called $
{service}-atlas-application.properties that will be saved to /etc/${service}
/conf/application.properties

These are the default values,

Falcon
atlas.hook.falcon.synchronous=false
atlas.hook.falcon.numRetries=3
atlas.hook.falcon.minThreads=5
atlas.hook.falcon.maxThreads=5
atlas.hook.falcon.keepAliveTime=10
atlas.hook.falcon.queueSize

Storm
atlas.hook.storm.numRetries=3

Hive
atlas.hook.hive.synchronous=false
atlas.hook.hive.numRetries=3
atlas.hook.hive.minThreads=5
atlas.hook.hive.maxThreads=5
atlas.hook.hive.keepAliveTime=10
atlas.hook.hive.queueSize=1

Common for all hooks
atlas.kafka.zookeeper.connect=
atlas.kafka.bootstrap.servers=
atlas.kafka.zookeeper.session.timeout.ms=400
atlas.kafka.zookeeper.connection.timeout.ms=200
atlas.kafka.zookeeper.sync.time.ms=20
atlas.kafka.hook.group.id=atlas
atlas.notification.create.topics=true
atlas.notification.replicas=1
atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
atlas.notification.kafka.service.principal=kafka/_h...@example.com
atlas.notification.kafka.keytab.location=/etc/security/keytabs/kafka.service.keytab
atlas.jaas.KafkaClient.loginModuleName = 
com.sun.security.auth.module.Krb5LoginModule
atlas.jaas.KafkaClient.loginModuleControlFlag = required
atlas.jaas.KafkaClient.option.useKeyTab = true
atlas.jaas.KafkaClient.option.storeKey = true
atlas.jaas.KafkaClient.option.serviceName = kafka
atlas.jaas.KafkaClient.option.keyTab = 
/etc/security/keytabs/atlas.service.keytab
atlas.jaas.KafkaClient.option.principal = atlas/_h...@example.com
atlas.cluster.name=


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/setup_atlas_hook.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
 1437251 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-atlas-application.properties.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/metainfo.xml 
602144b 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
 c2f1f53 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
 fc9d8b9 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py
 1dce515 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-atlas-application.properties.xml
 PRE-CREATION 
  ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 
273133a 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat.py
 839ab04 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
 ea2af62 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
 17f7380 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_atlas_hive.py
 d1bd8ea 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat.py
 816b6af 
  ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/metainfo.xml 
e3aa5ef 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py
 b2a6802 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/setup_atlas_sqoop.py
 76c1cda 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop.py
 bac836c 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py
 fac6331 
  
ambari-ser

Re: Review Request 49676: Add atlas-application config sections to all services that run Atlas hook, e.g., Hive, Falcon, Storm, Sqoop

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49676/#review141060
---


Ship it!




Ship It!

- Robert Levas


On July 6, 2016, 2:48 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49676/
> ---
> 
> (Updated July 6, 2016, 2:48 p.m.)
> 
> 
> Review request for Ambari, Madhan Neethiraj, Robert Levas, Sumit Mohanty, 
> Swapan Shridhar, and Suma Shivaprasad.
> 
> 
> Bugs: AMBARI-17573
> https://issues.apache.org/jira/browse/AMBARI-17573
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently, Atlas hooks that run in Hive, Falconm Storm, Sqoop processes 
> reference atlas-application.properties file from Atlas server config location 
> - /etc/atlas/conf/atlas-application.properties.
> Not all properties in /etc/atlas/conf/atlas-application.properties are 
> required in hooks and some of these properties are sensitive enough not to 
> expose them to hooks/clients.
> 
> To address this concern:
> 1. atlas-application.properties should be added as a config section in each 
> of the host component's that run Atlas hook - Hive, Storm, Falcon, Sqoop
> 2. These new config sections will only include properties that are required 
> to the respective hooks
> 3. During initial deployment, Ambari will initialize these properties with 
> values in Atlas server configuration.
> For each one of those services, create a config type called $
> {service}-atlas-application.properties that will be saved to /etc/${service}
> /conf/application.properties
> 
> These are the default values,
> 
> Falcon
> atlas.hook.falcon.synchronous=false
> atlas.hook.falcon.numRetries=3
> atlas.hook.falcon.minThreads=5
> atlas.hook.falcon.maxThreads=5
> atlas.hook.falcon.keepAliveTime=10
> atlas.hook.falcon.queueSize
> 
> Storm
> atlas.hook.storm.numRetries=3
> 
> Hive
> atlas.hook.hive.synchronous=false
> atlas.hook.hive.numRetries=3
> atlas.hook.hive.minThreads=5
> atlas.hook.hive.maxThreads=5
> atlas.hook.hive.keepAliveTime=10
> atlas.hook.hive.queueSize=1
> 
> Common for all hooks
> atlas.kafka.zookeeper.connect=
> atlas.kafka.bootstrap.servers=
> atlas.kafka.zookeeper.session.timeout.ms=400
> atlas.kafka.zookeeper.connection.timeout.ms=200
> atlas.kafka.zookeeper.sync.time.ms=20
> atlas.kafka.hook.group.id=atlas
> atlas.notification.create.topics=true
> atlas.notification.replicas=1
> atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
> atlas.notification.kafka.service.principal=kafka/_h...@example.com
> atlas.notification.kafka.keytab.location=/etc/security/keytabs/kafka.service.keytab
> atlas.jaas.KafkaClient.loginModuleName = 
> com.sun.security.auth.module.Krb5LoginModule
> atlas.jaas.KafkaClient.loginModuleControlFlag = required
> atlas.jaas.KafkaClient.option.useKeyTab = true
> atlas.jaas.KafkaClient.option.storeKey = true
> atlas.jaas.KafkaClient.option.serviceName = kafka
> atlas.jaas.KafkaClient.option.keyTab = 
> /etc/security/keytabs/atlas.service.keytab
> atlas.jaas.KafkaClient.option.principal = atlas/_h...@example.com
> atlas.cluster.name=
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_atlas_hook.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
>  1437251 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-atlas-application.properties.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/metainfo.xml
>  602144b 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
>  c2f1f53 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  fc9d8b9 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py
>  1dce515 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-atlas-application.properties.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 
> 273133a 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat.py
>  839ab04 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
>  ea2af62 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
>  17f7380 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_atlas_hive.py
>  d1bd8ea 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat.py

Re: Review Request 49676: Add atlas-application config sections to all services that run Atlas hook, e.g., Hive, Falcon, Storm, Sqoop

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49676/#review141057
---


Ship it!




Ship It!

- Nate Cole


On July 6, 2016, 2:48 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49676/
> ---
> 
> (Updated July 6, 2016, 2:48 p.m.)
> 
> 
> Review request for Ambari, Madhan Neethiraj, Robert Levas, Sumit Mohanty, 
> Swapan Shridhar, and Suma Shivaprasad.
> 
> 
> Bugs: AMBARI-17573
> https://issues.apache.org/jira/browse/AMBARI-17573
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently, Atlas hooks that run in Hive, Falconm Storm, Sqoop processes 
> reference atlas-application.properties file from Atlas server config location 
> - /etc/atlas/conf/atlas-application.properties.
> Not all properties in /etc/atlas/conf/atlas-application.properties are 
> required in hooks and some of these properties are sensitive enough not to 
> expose them to hooks/clients.
> 
> To address this concern:
> 1. atlas-application.properties should be added as a config section in each 
> of the host component's that run Atlas hook - Hive, Storm, Falcon, Sqoop
> 2. These new config sections will only include properties that are required 
> to the respective hooks
> 3. During initial deployment, Ambari will initialize these properties with 
> values in Atlas server configuration.
> For each one of those services, create a config type called $
> {service}-atlas-application.properties that will be saved to /etc/${service}
> /conf/application.properties
> 
> These are the default values,
> 
> Falcon
> atlas.hook.falcon.synchronous=false
> atlas.hook.falcon.numRetries=3
> atlas.hook.falcon.minThreads=5
> atlas.hook.falcon.maxThreads=5
> atlas.hook.falcon.keepAliveTime=10
> atlas.hook.falcon.queueSize
> 
> Storm
> atlas.hook.storm.numRetries=3
> 
> Hive
> atlas.hook.hive.synchronous=false
> atlas.hook.hive.numRetries=3
> atlas.hook.hive.minThreads=5
> atlas.hook.hive.maxThreads=5
> atlas.hook.hive.keepAliveTime=10
> atlas.hook.hive.queueSize=1
> 
> Common for all hooks
> atlas.kafka.zookeeper.connect=
> atlas.kafka.bootstrap.servers=
> atlas.kafka.zookeeper.session.timeout.ms=400
> atlas.kafka.zookeeper.connection.timeout.ms=200
> atlas.kafka.zookeeper.sync.time.ms=20
> atlas.kafka.hook.group.id=atlas
> atlas.notification.create.topics=true
> atlas.notification.replicas=1
> atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
> atlas.notification.kafka.service.principal=kafka/_h...@example.com
> atlas.notification.kafka.keytab.location=/etc/security/keytabs/kafka.service.keytab
> atlas.jaas.KafkaClient.loginModuleName = 
> com.sun.security.auth.module.Krb5LoginModule
> atlas.jaas.KafkaClient.loginModuleControlFlag = required
> atlas.jaas.KafkaClient.option.useKeyTab = true
> atlas.jaas.KafkaClient.option.storeKey = true
> atlas.jaas.KafkaClient.option.serviceName = kafka
> atlas.jaas.KafkaClient.option.keyTab = 
> /etc/security/keytabs/atlas.service.keytab
> atlas.jaas.KafkaClient.option.principal = atlas/_h...@example.com
> atlas.cluster.name=
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_atlas_hook.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
>  1437251 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-atlas-application.properties.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/metainfo.xml
>  602144b 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
>  c2f1f53 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  fc9d8b9 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py
>  1dce515 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-atlas-application.properties.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 
> 273133a 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat.py
>  839ab04 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
>  ea2af62 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
>  17f7380 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_atlas_hive.py
>  d1bd8ea 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat.py
> 

Re: Review Request 49676: Add atlas-application config sections to all services that run Atlas hook, e.g., Hive, Falcon, Storm, Sqoop

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49676/
---

(Updated July 6, 2016, 6:48 p.m.)


Review request for Ambari, Madhan Neethiraj, Robert Levas, Sumit Mohanty, 
Swapan Shridhar, and Suma Shivaprasad.


Changes
---

Addressed review comments


Bugs: AMBARI-17573
https://issues.apache.org/jira/browse/AMBARI-17573


Repository: ambari


Description
---

Currently, Atlas hooks that run in Hive, Falconm Storm, Sqoop processes 
reference atlas-application.properties file from Atlas server config location - 
/etc/atlas/conf/atlas-application.properties.
Not all properties in /etc/atlas/conf/atlas-application.properties are required 
in hooks and some of these properties are sensitive enough not to expose them 
to hooks/clients.

To address this concern:
1. atlas-application.properties should be added as a config section in each of 
the host component's that run Atlas hook - Hive, Storm, Falcon, Sqoop
2. These new config sections will only include properties that are required to 
the respective hooks
3. During initial deployment, Ambari will initialize these properties with 
values in Atlas server configuration.
For each one of those services, create a config type called $
{service}-atlas-application.properties that will be saved to /etc/${service}
/conf/application.properties

These are the default values,

Falcon
atlas.hook.falcon.synchronous=false
atlas.hook.falcon.numRetries=3
atlas.hook.falcon.minThreads=5
atlas.hook.falcon.maxThreads=5
atlas.hook.falcon.keepAliveTime=10
atlas.hook.falcon.queueSize

Storm
atlas.hook.storm.numRetries=3

Hive
atlas.hook.hive.synchronous=false
atlas.hook.hive.numRetries=3
atlas.hook.hive.minThreads=5
atlas.hook.hive.maxThreads=5
atlas.hook.hive.keepAliveTime=10
atlas.hook.hive.queueSize=1

Common for all hooks
atlas.kafka.zookeeper.connect=
atlas.kafka.bootstrap.servers=
atlas.kafka.zookeeper.session.timeout.ms=400
atlas.kafka.zookeeper.connection.timeout.ms=200
atlas.kafka.zookeeper.sync.time.ms=20
atlas.kafka.hook.group.id=atlas
atlas.notification.create.topics=true
atlas.notification.replicas=1
atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
atlas.notification.kafka.service.principal=kafka/_h...@example.com
atlas.notification.kafka.keytab.location=/etc/security/keytabs/kafka.service.keytab
atlas.jaas.KafkaClient.loginModuleName = 
com.sun.security.auth.module.Krb5LoginModule
atlas.jaas.KafkaClient.loginModuleControlFlag = required
atlas.jaas.KafkaClient.option.useKeyTab = true
atlas.jaas.KafkaClient.option.storeKey = true
atlas.jaas.KafkaClient.option.serviceName = kafka
atlas.jaas.KafkaClient.option.keyTab = 
/etc/security/keytabs/atlas.service.keytab
atlas.jaas.KafkaClient.option.principal = atlas/_h...@example.com
atlas.cluster.name=


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/setup_atlas_hook.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
 1437251 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-atlas-application.properties.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/metainfo.xml 
602144b 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
 c2f1f53 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
 fc9d8b9 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py
 1dce515 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-atlas-application.properties.xml
 PRE-CREATION 
  ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 
273133a 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat.py
 839ab04 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
 ea2af62 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
 17f7380 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_atlas_hive.py
 d1bd8ea 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat.py
 816b6af 
  ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/metainfo.xml 
e3aa5ef 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py
 b2a6802 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/setup_atlas_sqoop.py
 76c1cda 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop.py
 bac836c 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py
 fac6331

Re: Review Request 49676: Add atlas-application config sections to all services that run Atlas hook, e.g., Hive, Falcon, Storm, Sqoop

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49676/#review140986
---


Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/setup_atlas_hook.py
 (line 34)


Seems like this should be made _public_ so that it can be reused.



ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
 (lines 511 - 512)


You can reuse `has_atlas_in_cluster` from `setup_atlas_hook.py` (if made 
_public_)



ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py
 (lines 217 - 218)


You can reuse `has_atlas_in_cluster` from `setup_atlas_hook.py` (if made 
_public_)


- Robert Levas


On July 5, 2016, 9:13 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49676/
> ---
> 
> (Updated July 5, 2016, 9:13 p.m.)
> 
> 
> Review request for Ambari, Madhan Neethiraj, Robert Levas, Sumit Mohanty, 
> Swapan Shridhar, and Suma Shivaprasad.
> 
> 
> Bugs: AMBARI-17573
> https://issues.apache.org/jira/browse/AMBARI-17573
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently, Atlas hooks that run in Hive, Falconm Storm, Sqoop processes 
> reference atlas-application.properties file from Atlas server config location 
> - /etc/atlas/conf/atlas-application.properties.
> Not all properties in /etc/atlas/conf/atlas-application.properties are 
> required in hooks and some of these properties are sensitive enough not to 
> expose them to hooks/clients.
> 
> To address this concern:
> 1. atlas-application.properties should be added as a config section in each 
> of the host component's that run Atlas hook - Hive, Storm, Falcon, Sqoop
> 2. These new config sections will only include properties that are required 
> to the respective hooks
> 3. During initial deployment, Ambari will initialize these properties with 
> values in Atlas server configuration.
> For each one of those services, create a config type called $
> {service}-atlas-application.properties that will be saved to /etc/${service}
> /conf/application.properties
> 
> These are the default values,
> 
> Falcon
> atlas.hook.falcon.synchronous=false
> atlas.hook.falcon.numRetries=3
> atlas.hook.falcon.minThreads=5
> atlas.hook.falcon.maxThreads=5
> atlas.hook.falcon.keepAliveTime=10
> atlas.hook.falcon.queueSize
> 
> Storm
> atlas.hook.storm.numRetries=3
> 
> Hive
> atlas.hook.hive.synchronous=false
> atlas.hook.hive.numRetries=3
> atlas.hook.hive.minThreads=5
> atlas.hook.hive.maxThreads=5
> atlas.hook.hive.keepAliveTime=10
> atlas.hook.hive.queueSize=1
> 
> Common for all hooks
> atlas.kafka.zookeeper.connect=
> atlas.kafka.bootstrap.servers=
> atlas.kafka.zookeeper.session.timeout.ms=400
> atlas.kafka.zookeeper.connection.timeout.ms=200
> atlas.kafka.zookeeper.sync.time.ms=20
> atlas.kafka.hook.group.id=atlas
> atlas.notification.create.topics=true
> atlas.notification.replicas=1
> atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
> atlas.notification.kafka.service.principal=kafka/_h...@example.com
> atlas.notification.kafka.keytab.location=/etc/security/keytabs/kafka.service.keytab
> atlas.jaas.KafkaClient.loginModuleName = 
> com.sun.security.auth.module.Krb5LoginModule
> atlas.jaas.KafkaClient.loginModuleControlFlag = required
> atlas.jaas.KafkaClient.option.useKeyTab = true
> atlas.jaas.KafkaClient.option.storeKey = true
> atlas.jaas.KafkaClient.option.serviceName = kafka
> atlas.jaas.KafkaClient.option.keyTab = 
> /etc/security/keytabs/atlas.service.keytab
> atlas.jaas.KafkaClient.option.principal = atlas/_h...@example.com
> atlas.cluster.name=
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_atlas_hook.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
>  1437251 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-atlas-application.properties.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/metainfo.xml
>  602144b 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
>  c2f1f53 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  fc9d8b9 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py
>  1dce515 
>   
> ambari-server/src/main/resources/common-services

Review Request 49676: Add atlas-application config sections to all services that run Atlas hook, e.g., Hive, Falcon, Storm, Sqoop

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49676/
---

Review request for Ambari, Robert Levas, Sumit Mohanty, and Swapan Shridhar.


Bugs: AMBARI-17573
https://issues.apache.org/jira/browse/AMBARI-17573


Repository: ambari


Description
---

Currently, Atlas hooks that run in Hive, Falconm Storm, Sqoop processes 
reference atlas-application.properties file from Atlas server config location - 
/etc/atlas/conf/atlas-application.properties.
Not all properties in /etc/atlas/conf/atlas-application.properties are required 
in hooks and some of these properties are sensitive enough not to expose them 
to hooks/clients.

To address this concern:
1. atlas-application.properties should be added as a config section in each of 
the host component's that run Atlas hook - Hive, Storm, Falcon, Sqoop
2. These new config sections will only include properties that are required to 
the respective hooks
3. During initial deployment, Ambari will initialize these properties with 
values in Atlas server configuration.
For each one of those services, create a config type called $
{service}-atlas-application.properties that will be saved to /etc/${service}
/conf/application.properties

These are the default values,

Falcon
atlas.hook.falcon.synchronous=false
atlas.hook.falcon.numRetries=3
atlas.hook.falcon.minThreads=5
atlas.hook.falcon.maxThreads=5
atlas.hook.falcon.keepAliveTime=10
atlas.hook.falcon.queueSize

Storm
atlas.hook.storm.numRetries=3

Hive
atlas.hook.hive.synchronous=false
atlas.hook.hive.numRetries=3
atlas.hook.hive.minThreads=5
atlas.hook.hive.maxThreads=5
atlas.hook.hive.keepAliveTime=10
atlas.hook.hive.queueSize=1

Common for all hooks
atlas.kafka.zookeeper.connect=
atlas.kafka.bootstrap.servers=
atlas.kafka.zookeeper.session.timeout.ms=400
atlas.kafka.zookeeper.connection.timeout.ms=200
atlas.kafka.zookeeper.sync.time.ms=20
atlas.kafka.hook.group.id=atlas
atlas.notification.create.topics=true
atlas.notification.replicas=1
atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
atlas.notification.kafka.service.principal=kafka/_h...@example.com
atlas.notification.kafka.keytab.location=/etc/security/keytabs/kafka.service.keytab
atlas.jaas.KafkaClient.loginModuleName = 
com.sun.security.auth.module.Krb5LoginModule
atlas.jaas.KafkaClient.loginModuleControlFlag = required
atlas.jaas.KafkaClient.option.useKeyTab = true
atlas.jaas.KafkaClient.option.storeKey = true
atlas.jaas.KafkaClient.option.serviceName = kafka
atlas.jaas.KafkaClient.option.keyTab = 
/etc/security/keytabs/atlas.service.keytab
atlas.jaas.KafkaClient.option.principal = atlas/_h...@example.com
atlas.cluster.name=


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/setup_atlas_hook.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
 1437251 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-atlas-application.properties.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/metainfo.xml 
602144b 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
 c2f1f53 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
 fc9d8b9 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py
 1dce515 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-atlas-application.properties.xml
 PRE-CREATION 
  ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 
273133a 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat.py
 839ab04 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
 ea2af62 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
 17f7380 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_atlas_hive.py
 d1bd8ea 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat.py
 816b6af 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py
 b2a6802 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/setup_atlas_sqoop.py
 76c1cda 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop.py
 bac836c 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py
 fac6331 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/setup_atlas_storm.py
 76ca1d3 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/storm.py
 2d50767 
  
amb