> On July 9, 2019, 10:55 p.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I 
> > don't see a need for configuration "atlas.metadata.namespace". I suggest 
> > that we just use the <clustername>.<servicename> format. It is hard to 
> > manage with configurations like this. What would happen if this 
> > configuration is changed? What will you do with existing objects when this 
> > value is changed. This needs some kind of design exercise. Can we please 
> > have a . meeting for it.
> 
> Madhan Neethiraj wrote:
>     Sridhar - current use of 'clusterName' works when all services that 
> access (CRUD) a resource run in the same cluster. There are cases where 
> services running in a cluster that is different from where the resource 
> resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services 
> running in both clusters can access Hive metastore running in "cl1". In such 
> scenarios holding on to 'clusterName' would be confusing; hence the move to a 
> generic name 'metadata.namespace'. It makes it intutive to ask the services 
> set this configuration, instead of confusing them with 'clusterName'. Hope 
> this helps.
> 
> Sridhar K wrote:
>     I see your point. But, the issue is in the possibility of wrongly 
> configured names and flexibility of changing the names. In the example you 
> gave, I would expect HMS to have the convention of "cl1.hms1" as auto 
> generated configuration and Hive-Server resources like hive_processes should 
> have their name spaces as "cl1.hive-server-1" and " cl2.hive-server-1". I 
> think that this gives a general understanding on namespace hierarchy. If it 
> is user configurable, user might accidentally give the same name for every 
> cluster resulting in mixups. So, I think that we need to give some thinking. 
> You might have given some thinking on it. If so,  I would like to understand 
> it.
> 
> Sridhar K wrote:
>     If we allow the user to give values for namespace, we might have to have 
> this information captured as first class Atlas Entity as well--for auditing, 
> goveranance point of view.

Sridhar, 
metadata_namespace will eventually replace cluster name and going forward atlas 
will be using metadata namespace instead of cluster name to construct qualified 
name. This patch is more of replacing cluster name to metadata namespace name.


- Sarath


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
-----------------------------------------------------------


On July 9, 2019, 4:53 p.m., Sarath Subramanian wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
> 
> (Updated July 9, 2019, 4:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
>     https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -----
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
> 
> 
> Diff: https://reviews.apache.org/r/71042/diff/1/
> 
> 
> Testing
> -------
> 
> Testing in progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Reply via email to