> On Sept. 21, 2016, 8:58 p.m., Suma Shivaprasad wrote: > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java, > > line 97 > > <https://reviews.apache.org/r/52077/diff/2/?file=1505617#file1505617line97> > > > > why is column qualifiedName different from the convention we are using > > for hive_column instances which are referred to from the table. Why is > > clustername removed? > > Vimal Sharma wrote: > Cluster information is not available in Lineage information provided by > Hive. Further, qualifiedName used in this patch is used only while setting > column lineage and is not used for communication with rest of Atlas codebase. > > Suma Shivaprasad wrote: > If we do not provide the same qualifiedName as in the current > HMSB.getColumnQualifiedName() , it will result in a another entity being > created for the columns. Cluster information is available in > HMSB.getClusterName() > > Vimal Sharma wrote: > In the function populateColumnReferenceableMap, we are setting a mapping > from column string identifier(named as column qualified name) to its > corresponding column Referenceable object in Atlas. No new column > Referenceable entity is created. > > Further, in buildLineageMap, we are setting a mapping from destination > column qualified name to list of source column qualified names. Now, in the > key value pairs of the type (LineageInfo.DependencyKey, > LineageInfo.Dependency) in LineageInfo from Hive, there is no cluster > information available. So here we can't use the same pattern for column > qualified name as used in HMSB.getColumnQualifiedName. > > If we set column string identifier as HMSB.getColumnQualifiedName in > function populateColumnReferenceableMap, we won't be able to access the > column referenceable objects from the map(created in > populateColumnReferenceableMap) in HiveHook when we are setting up column > lineage process in function createColumnLineageProcessInstances(lines 803 and > 812).
Sounds good - Suma ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52077/#review149896 ----------------------------------------------------------- On Sept. 29, 2016, 8:15 a.m., Vimal Sharma wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52077/ > ----------------------------------------------------------- > > (Updated Sept. 29, 2016, 8:15 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-1184 and ATLAS-247 > https://issues.apache.org/jira/browse/ATLAS-1184 > https://issues.apache.org/jira/browse/ATLAS-247 > > > Repository: atlas > > > Description > ------- > > After a CTAS query, lineage relationship between source columns and > destination column can be captured. This information can be used to create a > column lineage process. > > > Diffs > ----- > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java > PRE-CREATION > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > a3464a0 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java > 45f0bc9 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java > e094cb6 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > a5838b4 > > Diff: https://reviews.apache.org/r/52077/diff/ > > > Testing > ------- > > > Thanks, > > Vimal Sharma > >
