> On July 10, 2019, 5:55 a.m., Sridhar K wrote: > > Hello Sarath, > > Is this some thing new that we are doing? I understand the reasoning. I > > don't see a need for configuration "atlas.metadata.namespace". I suggest > > that we just use the <clustername>.<servicename> format. It is hard to > > manage with configurations like this. What would happen if this > > configuration is changed? What will you do with existing objects when this > > value is changed. This needs some kind of design exercise. Can we please > > have a . meeting for it. > > Madhan Neethiraj wrote: > Sridhar - current use of 'clusterName' works when all services that > access (CRUD) a resource run in the same cluster. There are cases where > services running in a cluster that is different from where the resource > resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services > running in both clusters can access Hive metastore running in "cl1". In such > scenarios holding on to 'clusterName' would be confusing; hence the move to a > generic name 'metadata.namespace'. It makes it intutive to ask the services > set this configuration, instead of confusing them with 'clusterName'. Hope > this helps. > > Sridhar K wrote: > I see your point. But, the issue is in the possibility of wrongly > configured names and flexibility of changing the names. In the example you > gave, I would expect HMS to have the convention of "cl1.hms1" as auto > generated configuration and Hive-Server resources like hive_processes should > have their name spaces as "cl1.hive-server-1" and " cl2.hive-server-1". I > think that this gives a general understanding on namespace hierarchy. If it > is user configurable, user might accidentally give the same name for every > cluster resulting in mixups. So, I think that we need to give some thinking. > You might have given some thinking on it. If so, I would like to understand > it.
If we allow the user to give values for namespace, we might have to have this information captured as first class Atlas Entity as well--for auditing, goveranance point of view. - Sridhar ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71042/#review216470 ----------------------------------------------------------- On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71042/ > ----------------------------------------------------------- > > (Updated July 9, 2019, 11:53 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Nixon Rodrigues. > > > Bugs: ATLAS-3321 > https://issues.apache.org/jira/browse/ATLAS-3321 > > > Repository: atlas > > > Description > ------- > > Atlas has the notion of clusterName which is used as a global namespace in > which you can define the context in which entity instances are defined and > grouped. A cluster name defines a scope of operation in atlas and is > specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), > fs_path (File System), storm_topology (Storm), spark_db (Spark) > > cluster name is also used to construct qualifiedName attribute in atlas - > > hive_db - db_name@cluster_name > hive_table - db_name.table_name@cluster_name > hive_column - db_name.table_name.column_name@cluster_name > Cluster name works for simple environments wherein we have a single cluster > and single service instances running (single Hive or HBase service). In > complex deployments - a single cluster may host multiple instances of the > same service (two Hive services - Hive-1 and Hive-2), for such cases cluster > name doesn't help to provide context to individual service instances and > metadata groupings. > > So we introduce - "metadata namespace" where a single cluster can contain > multiple metadata namespaces. The qualified name for atlas will be generated > using metadata namespace name instead of cluster name. > > For each hook enabled we specify the property - "atlas.metadata.namespace" > which defines the scope/context of the metadata. > > > Diffs > ----- > > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java > 1825cd290 > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java > 1765c1848 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 5f8f846ae > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java > 76d6fe612 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > ffa56cea0 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > 05d79df29 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java > cbee7bf14 > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java > d55aa5370 > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java > 1305f65a6 > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java > b5fdb6dbd > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java > 4ea484f52 > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > f1d02375c > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > 8ebb385cf > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > a7b9b0cd0 > > addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java > 8755c9e60 > > addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java > 5397a4b0b > > addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java > 97668a3c6 > common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d > > > Diff: https://reviews.apache.org/r/71042/diff/1/ > > > Testing > ------- > > Testing in progress > > > Thanks, > > Sarath Subramanian > >
