> On Sept. 21, 2016, 8:58 p.m., Suma Shivaprasad wrote: > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java, > > line 97 > > <https://reviews.apache.org/r/52077/diff/2/?file=1505617#file1505617line97> > > > > why is column qualifiedName different from the convention we are using > > for hive_column instances which are referred to from the table. Why is > > clustername removed? > > Vimal Sharma wrote: > Cluster information is not available in Lineage information provided by > Hive. Further, qualifiedName used in this patch is used only while setting > column lineage and is not used for communication with rest of Atlas codebase. > > Suma Shivaprasad wrote: > If we do not provide the same qualifiedName as in the current > HMSB.getColumnQualifiedName() , it will result in a another entity being > created for the columns. Cluster information is available in > HMSB.getClusterName()
In the function populateColumnReferenceableMap, we are setting a mapping from column string identifier(named as column qualified name) to its corresponding column Referenceable object in Atlas. No new column Referenceable entity is created. Further, in buildLineageMap, we are setting a mapping from destination column qualified name to list of source column qualified names. Now, in the key value pairs of the type (LineageInfo.DependencyKey, LineageInfo.Dependency) in LineageInfo from Hive, there is no cluster information available. So here we can't use the same pattern for column qualified name as used in HMSB.getColumnQualifiedName. If we set column string identifier as HMSB.getColumnQualifiedName in function populateColumnReferenceableMap, we won't be able to access the column referenceable objects from the map(created in populateColumnReferenceableMap) in HiveHook when we are setting up column lineage process in function createColumnLineageProcessInstances(lines 803 and 812). - Vimal ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52077/#review149896 ----------------------------------------------------------- On Sept. 22, 2016, 11:53 a.m., Vimal Sharma wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52077/ > ----------------------------------------------------------- > > (Updated Sept. 22, 2016, 11:53 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-247 > https://issues.apache.org/jira/browse/ATLAS-247 > > > Repository: atlas > > > Description > ------- > > After a CTAS query, lineage relationship between source columns and > destination column can be captured. This information can be used to create a > column lineage process. > > > Diffs > ----- > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java > PRE-CREATION > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > a3464a0 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java > 45f0bc9 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java > e094cb6 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > a5838b4 > > Diff: https://reviews.apache.org/r/52077/diff/ > > > Testing > ------- > > > Thanks, > > Vimal Sharma > >
