----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71902/#review219011 -----------------------------------------------------------
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java Lines 66 (patched) <https://reviews.apache.org/r/71902/#comment307029> LINEAGE_GRAPH_QUERY("atlas.use.graph.query.for.lineage", false) => LINEAGE_USING_GREMLIN("atlas.lineage.query.use.gremlin", false) repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Lines 84 (patched) <https://reviews.apache.org/r/71902/#comment307030> useGraphQueryForLineage => LINEAGE_USING_GREMLIN repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Lines 205 (patched) <https://reviews.apache.org/r/71902/#comment307023> To be consistent, consider replacing "UsingGraphQuery" with "V1" as suffix. repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Lines 264 (patched) <https://reviews.apache.org/r/71902/#comment307034> Consider returning "Collection<AtlasEdge>" from this method, as the caller uses only the value of the returned map. repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Lines 280 (patched) <https://reviews.apache.org/r/71902/#comment307025> How is 'direction == BOTH' handled here? Looks like only PROCESS_OUTPUTS_EDGE edges will be traversed. Please review. AtlasVertex processVertex = AtlasGraphUtilsV2.findByGuid(guid); if (direction == INPUT || direction == BOTH) { Iterable<AtlasEdge> processEdges = processVertex.getEdges(AtlasEdgeDirection.OUT, PROCESS_INPUTS_EDGE); for (AtlasEdge processEdge : processEdges) { AtlasVertex datasetVertex = processEdge.getInVertex(); traverseEdges(datasetVertex, INPUT, depth - 1, ret); } } if (direction == OUTPUT || direction == BOTH) { Iterable<AtlasEdge> processEdges = processVertex.getEdges(AtlasEdgeDirection.OUT, PROCESS_OUTPUTS_EDGE); for (AtlasEdge processEdge : processEdges) { AtlasVertex datasetVertex = processEdge.getInVertex(); traverseEdges(datasetVertex, OUTPUT, depth - 1, ret); } } repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Lines 303 (patched) <https://reviews.apache.org/r/71902/#comment307031> visitedIds => visitedVertices repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Lines 308 (patched) <https://reviews.apache.org/r/71902/#comment307028> Given 'BOTH' is not expected here, perhaps the parameter can be: "LineageDirection direction" => "boolean isInput" Same in #299 as well. repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Lines 312 (patched) <https://reviews.apache.org/r/71902/#comment307027> - can 'incomingEdge' be already present in 'ret'? If yes, rest of this loop body can be skipped with a 'continue' at #312. Please review. - can 'processVertex' be already visited? If yes, rest of this loop body can be skipped with a 'continue' at #313. For this to work, processVertex should be added to 'visitedIds'. Please review. repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Lines 320 (patched) <https://reviews.apache.org/r/71902/#comment307033> Looks like #320 can be moved before entering this 'for' loop, at #315. Please review. repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Lines 340 (patched) <https://reviews.apache.org/r/71902/#comment307024> startVertex => vertex - Madhan Neethiraj On Dec. 11, 2019, 11:59 p.m., Sarath Subramanian wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71902/ > ----------------------------------------------------------- > > (Updated Dec. 11, 2019, 11:59 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, > Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Nixon Rodrigues, Sameer > Shaikh, and Sarath Subramanian. > > > Bugs: ATLAS-3558 > https://issues.apache.org/jira/browse/ATLAS-3558 > > > Repository: atlas > > > Description > ------- > > Lineage in atlas uses graph query to compute lineage across entities (inputs, > outputs or both). Lineage rendering performance have degraded after using > janusgraph version 0.4.0 > > On investigation, lineage graph query initialization and execution using > gremlin script engine has been found to be the bottleneck. > > Alternate in-memory computation of lineage has improved performance by many > folds (~90% improvement). This Jira is about adding alternate in-memory > computation of lineage. > > "atlas.use.graph.query.for.lineage" property can be used to toggle between > graph query and in-memory computation of lineage. The default option will be > in-memory. > > > Diffs > ----- > > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 979bd0ae3 > > repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java > 9a020468d > > > Diff: https://reviews.apache.org/r/71902/diff/1/ > > > Testing > ------- > > Precommit: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1581/console > > Manually validated lineage rendering works fine for simple, complex and > circular lineages. > > > Thanks, > > Sarath Subramanian > >
