Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-11 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216541
---


Ship it!




Ship It!

- Madhan Neethiraj


On July 12, 2019, 1:30 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 12, 2019, 1:30 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   addons/hbase-bridge/src/bin/import-hbase.sh b9c1a4bc0 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   addons/hive-bridge/src/bin/import-hive.sh 38ec3be24 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  b3663da34 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   addons/kafka-bridge/src/bin/import-kafka.sh 4bfb8b96b 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 0030276c6 
> 
> 
> Diff: https://reviews.apache.org/r/71042/diff/3/
> 
> 
> Testing
> ---
> 
> Testing in progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-11 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/
---

(Updated July 11, 2019, 6:30 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Nixon Rodrigues.


Bugs: ATLAS-3321
https://issues.apache.org/jira/browse/ATLAS-3321


Repository: atlas


Description
---

Atlas has the notion of clusterName which is used as a global namespace in 
which you can define the context in which entity instances are defined and 
grouped. A cluster name defines a scope of operation in atlas and is specified 
in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File 
System), storm_topology (Storm), spark_db (Spark)

cluster name is also used to construct qualifiedName attribute in atlas -

hive_db - db_name@cluster_name
hive_table - db_name.table_name@cluster_name
hive_column - db_name.table_name.column_name@cluster_name
Cluster name works for simple environments wherein we have a single cluster and 
single service instances running (single Hive or HBase service). In complex 
deployments - a single cluster may host multiple instances of the same service 
(two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't 
help to provide context to individual service instances and metadata groupings.

So we introduce - "metadata namespace" where a single cluster can contain 
multiple metadata namespaces. The qualified name for atlas will be generated 
using metadata namespace name instead of cluster name.

For each hook enabled we specify the property - "atlas.metadata.namespace" 
which defines the scope/context of the metadata.


Diffs (updated)
-

  addons/hbase-bridge/src/bin/import-hbase.sh b9c1a4bc0 
  
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
 1825cd290 
  
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
 1765c1848 
  addons/hive-bridge/src/bin/import-hive.sh 38ec3be24 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 5f8f846ae 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
 76d6fe612 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
ffa56cea0 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 b3663da34 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
cbee7bf14 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
 d55aa5370 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
 1305f65a6 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 b5fdb6dbd 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 4ea484f52 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 f1d02375c 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 8ebb385cf 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 a7b9b0cd0 
  addons/kafka-bridge/src/bin/import-kafka.sh 4bfb8b96b 
  
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
 8755c9e60 
  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
5397a4b0b 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 97668a3c6 
  common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
  notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 0030276c6 


Diff: https://reviews.apache.org/r/71042/diff/3/

Changes: https://reviews.apache.org/r/71042/diff/2-3/


Testing
---

Testing in progress


Thanks,

Sarath Subramanian

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-10 Thread Sridhar K 


> On July 10, 2019, 10:12 p.m., Madhan Neethiraj wrote:
> > Ship It!

I am not convinved that this is correct approach.


- Sridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216503
---


On July 10, 2019, 8:11 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 10, 2019, 8:11 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  b3663da34 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 0030276c6 
> 
> 
> Diff: https://reviews.apache.org/r/71042/diff/2/
> 
> 
> Testing
> ---
> 
> Testing in progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-10 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/
---

(Updated July 10, 2019, 1:11 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Nixon Rodrigues.


Changes
---

addressed review comments.


Bugs: ATLAS-3321
https://issues.apache.org/jira/browse/ATLAS-3321


Repository: atlas


Description
---

Atlas has the notion of clusterName which is used as a global namespace in 
which you can define the context in which entity instances are defined and 
grouped. A cluster name defines a scope of operation in atlas and is specified 
in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File 
System), storm_topology (Storm), spark_db (Spark)

cluster name is also used to construct qualifiedName attribute in atlas -

hive_db - db_name@cluster_name
hive_table - db_name.table_name@cluster_name
hive_column - db_name.table_name.column_name@cluster_name
Cluster name works for simple environments wherein we have a single cluster and 
single service instances running (single Hive or HBase service). In complex 
deployments - a single cluster may host multiple instances of the same service 
(two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't 
help to provide context to individual service instances and metadata groupings.

So we introduce - "metadata namespace" where a single cluster can contain 
multiple metadata namespaces. The qualified name for atlas will be generated 
using metadata namespace name instead of cluster name.

For each hook enabled we specify the property - "atlas.metadata.namespace" 
which defines the scope/context of the metadata.


Diffs (updated)
-

  
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
 1825cd290 
  
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
 1765c1848 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 5f8f846ae 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
 76d6fe612 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
ffa56cea0 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 b3663da34 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
cbee7bf14 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
 d55aa5370 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
 1305f65a6 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 b5fdb6dbd 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 4ea484f52 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 f1d02375c 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 8ebb385cf 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 a7b9b0cd0 
  
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
 8755c9e60 
  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
5397a4b0b 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 97668a3c6 
  common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
  notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 0030276c6 


Diff: https://reviews.apache.org/r/71042/diff/2/

Changes: https://reviews.apache.org/r/71042/diff/1-2/


Testing
---

Testing in progress


Thanks,

Sarath Subramanian

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-10 Thread Sarath Subramanian 


> On July 9, 2019, 9:45 p.m., Ashutosh Mestry wrote:
> > addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
> > Lines 176 (patched)
> > 
> >
> > Duplicate code.

All the bridge code - HiveMetastoreBridge, HBaseBridge and KafkaBridge do not 
share a common base class to move the method. Hence dupliccated.


- Sarath


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216468
---


On July 9, 2019, 4:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 4:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
> 
> 
> Diff: https://reviews.apache.org/r/71042/diff/1/
> 
> 
> Testing
> ---
> 
> Testing in progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-10 Thread Sarath Subramanian 


> On July 9, 2019, 10:55 p.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I 
> > don't see a need for configuration "atlas.metadata.namespace". I suggest 
> > that we just use the . format. It is hard to 
> > manage with configurations like this. What would happen if this 
> > configuration is changed? What will you do with existing objects when this 
> > value is changed. This needs some kind of design exercise. Can we please 
> > have a . meeting for it.
> 
> Madhan Neethiraj wrote:
> Sridhar - current use of 'clusterName' works when all services that 
> access (CRUD) a resource run in the same cluster. There are cases where 
> services running in a cluster that is different from where the resource 
> resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services 
> running in both clusters can access Hive metastore running in "cl1". In such 
> scenarios holding on to 'clusterName' would be confusing; hence the move to a 
> generic name 'metadata.namespace'. It makes it intutive to ask the services 
> set this configuration, instead of confusing them with 'clusterName'. Hope 
> this helps.
> 
> Sridhar K wrote:
> I see your point. But, the issue is in the possibility of wrongly 
> configured names and flexibility of changing the names. In the example you 
> gave, I would expect HMS to have the convention of "cl1.hms1" as auto 
> generated configuration and Hive-Server resources like hive_processes should 
> have their name spaces as "cl1.hive-server-1" and " cl2.hive-server-1". I 
> think that this gives a general understanding on namespace hierarchy. If it 
> is user configurable, user might accidentally give the same name for every 
> cluster resulting in mixups. So, I think that we need to give some thinking. 
> You might have given some thinking on it. If so,  I would like to understand 
> it.
> 
> Sridhar K wrote:
> If we allow the user to give values for namespace, we might have to have 
> this information captured as first class Atlas Entity as well--for auditing, 
> goveranance point of view.

Sridhar, 
metadata_namespace will eventually replace cluster name and going forward atlas 
will be using metadata namespace instead of cluster name to construct qualified 
name. This patch is more of replacing cluster name to metadata namespace name.


- Sarath


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
---


On July 9, 2019, 4:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 4:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
>

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-10 Thread Sridhar K 


> On July 10, 2019, 5:55 a.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I 
> > don't see a need for configuration "atlas.metadata.namespace". I suggest 
> > that we just use the . format. It is hard to 
> > manage with configurations like this. What would happen if this 
> > configuration is changed? What will you do with existing objects when this 
> > value is changed. This needs some kind of design exercise. Can we please 
> > have a . meeting for it.
> 
> Madhan Neethiraj wrote:
> Sridhar - current use of 'clusterName' works when all services that 
> access (CRUD) a resource run in the same cluster. There are cases where 
> services running in a cluster that is different from where the resource 
> resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services 
> running in both clusters can access Hive metastore running in "cl1". In such 
> scenarios holding on to 'clusterName' would be confusing; hence the move to a 
> generic name 'metadata.namespace'. It makes it intutive to ask the services 
> set this configuration, instead of confusing them with 'clusterName'. Hope 
> this helps.
> 
> Sridhar K wrote:
> I see your point. But, the issue is in the possibility of wrongly 
> configured names and flexibility of changing the names. In the example you 
> gave, I would expect HMS to have the convention of "cl1.hms1" as auto 
> generated configuration and Hive-Server resources like hive_processes should 
> have their name spaces as "cl1.hive-server-1" and " cl2.hive-server-1". I 
> think that this gives a general understanding on namespace hierarchy. If it 
> is user configurable, user might accidentally give the same name for every 
> cluster resulting in mixups. So, I think that we need to give some thinking. 
> You might have given some thinking on it. If so,  I would like to understand 
> it.

If we allow the user to give values for namespace, we might have to have this 
information captured as first class Atlas Entity as well--for auditing, 
goveranance point of view.


- Sridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
---


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-10 Thread Sridhar K 


> On July 10, 2019, 5:55 a.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I 
> > don't see a need for configuration "atlas.metadata.namespace". I suggest 
> > that we just use the . format. It is hard to 
> > manage with configurations like this. What would happen if this 
> > configuration is changed? What will you do with existing objects when this 
> > value is changed. This needs some kind of design exercise. Can we please 
> > have a . meeting for it.
> 
> Madhan Neethiraj wrote:
> Sridhar - current use of 'clusterName' works when all services that 
> access (CRUD) a resource run in the same cluster. There are cases where 
> services running in a cluster that is different from where the resource 
> resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services 
> running in both clusters can access Hive metastore running in "cl1". In such 
> scenarios holding on to 'clusterName' would be confusing; hence the move to a 
> generic name 'metadata.namespace'. It makes it intutive to ask the services 
> set this configuration, instead of confusing them with 'clusterName'. Hope 
> this helps.

I see your point. But, the issue is in the possibility of wrongly configured 
names and flexibility of changing the names. In the example you gave, I would 
expect HMS to have the convention of "cl1.hms1" as auto generated configuration 
and Hive-Server resources like hive_processes should have their name spaces as 
"cl1.hive-server-1" and " cl2.hive-server-1". I think that this gives a general 
understanding on namespace hierarchy. If it is user configurable, user might 
accidentally give the same name for every cluster resulting in mixups. So, I 
think that we need to give some thinking. You might have given some thinking on 
it. If so,  I would like to understand it.


- Sridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
---


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
>

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-10 Thread Madhan Neethiraj 


> On July 10, 2019, 5:55 a.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I 
> > don't see a need for configuration "atlas.metadata.namespace". I suggest 
> > that we just use the . format. It is hard to 
> > manage with configurations like this. What would happen if this 
> > configuration is changed? What will you do with existing objects when this 
> > value is changed. This needs some kind of design exercise. Can we please 
> > have a . meeting for it.

Sridhar - current use of 'clusterName' works when all services that access 
(CRUD) a resource run in the same cluster. There are cases where services 
running in a cluster that is different from where the resource resides. For 
example, consider 2 clusters cl1 and cl2. HiveServer2 services running in both 
clusters can access Hive metastore running in "cl1". In such scenarios holding 
on to 'clusterName' would be confusing; hence the move to a generic name 
'metadata.namespace'. It makes it intutive to ask the services set this 
configuration, instead of confusing them with 'clusterName'. Hope this helps.


- Madhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
---


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
>

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-09 Thread Sridhar K 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
---



Hello Sarath,
Is this some thing new that we are doing? I understand the reasoning. I don't 
see a need for configuration "atlas.metadata.namespace". I suggest that we just 
use the . format. It is hard to manage with 
configurations like this. What would happen if this configuration is changed? 
What will you do with existing objects when this value is changed. This needs 
some kind of design exercise. Can we please have a . meeting for it.

- Sridhar K


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
> 
> 
> Diff: https://reviews.apache.org/r/71042/diff/1/
> 
> 
> Testing
> ---
> 
> Testing in progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-09 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216469
---



Will make sense to write a unit test that exercises the logic for 
getMetadataNamespace.

- Ashutosh Mestry


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
> 
> 
> Diff: https://reviews.apache.org/r/71042/diff/1/
> 
> 
> Testing
> ---
> 
> Testing in progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-09 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216468
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
Line 256 (original), 260 (patched)


Duplicate code.



addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
Line 146 (original), 151 (patched)


Duplicate code.



addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Line 65 (original), 65 (patched)


Can be renamed to be made consistent with other hooks.



addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 176 (patched)


Duplicate code.


- Ashutosh Mestry


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
> 
> 
> Diff: https://reviews.apache.org/r/71042/diff/1/
> 
> 
> Testing
> ---
> 
> Testing in progress
> 
> 
> Thanks,
> 
> Sarath

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

2019-07-09 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216460
---




addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
Line 59 (original), 60 (patched)


Configurations "atlas.cluster.name", "atlas.metadata.namespace" and their 
default values - are better better handled in base class AtlasHook, to avoid 
having to deal with this in every hook. Please review and consider refactoring.



addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
Line 60 (original), 61 (patched)


I suggest to use earlier default value - "primary".



addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
Line 109 (original), 112 (patched)


Is it necessary to retain 'clusterName' field?

Please review clusterName in other classes as well, like:
- HBaseBridge
- HiveHook
- ImpalaLineageHook


- Madhan Neethiraj


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>