> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote: > > addons/impala-bridge-shim/pom.xml > > Lines 28 (patched) > > <https://reviews.apache.org/r/70512/diff/13/?file=2142459#file2142459line28> > > > > shim is not needed; since this will be used as a standalone utility.
removed > On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java > > Lines 35 (patched) > > <https://reviews.apache.org/r/70512/diff/13/?file=2142463#file2142463line35> > > > > TEMP_TABLE_PREFIX is never used; consider removing it. removed > On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java > > Lines 65 (patched) > > <https://reviews.apache.org/r/70512/diff/13/?file=2142464#file2142464line65> > > > > return after LOG.warn stmt; might hit NPE in line #73 fixed > On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java > > Lines 33 (patched) > > <https://reviews.apache.org/r/70512/diff/13/?file=2142465#file2142465line33> > > > > why the following queries are not handled? > > * CREATE TABLE > > * INSERT INTO TABLE <table_name> SELECT > > * INSERT OVERWRITE TABLE This patch only supports "create view" command. More command support will be done in ATLAS-3184. The goal of this patch is to create Impala-bridge project and get integration work end-to-end. > On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > > Lines 459 (patched) > > <https://reviews.apache.org/r/70512/diff/13/?file=2142466#file2142466line459> > > > > columnPosition never used? consider removing it. removed > On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > > Lines 490 (patched) > > <https://reviews.apache.org/r/70512/diff/13/?file=2142466#file2142466line490> > > > > can we have new model change to create "impala_process" typedef with > > supertype "process" as part of this patch? Will be good to see > > impala_process connecting hive entities in the server side. added types for processes. Reuse table, db types public enum ImpalaDataType { IMPALA_PROCESS, IMPALA_PROCESS_EXECUTION, IMPALA_COLUMN_LINEAGE; public String getName() { return name().toLowerCase(); } } > On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > > Lines 71 (patched) > > <https://reviews.apache.org/r/70512/diff/13/?file=2142467#file2142467line71> > > > > what cases do we want to skip impala_process creation? I changed the code. get inputs and outputs first. If one of them is empty, skip the process. > On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > > Lines 133 (patched) > > <https://reviews.apache.org/r/70512/diff/13/?file=2142467#file2142467line133> > > > > remove edges==null check; CollectionUtils.isEmpty(edges) does null > > check as well. removed > On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > > Lines 138 (patched) > > <https://reviews.apache.org/r/70512/diff/13/?file=2142467#file2142467line138> > > > > lineageInputsCount is unused; consider removing if not used. removed - Na ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70512/#review215013 ----------------------------------------------------------- On May 2, 2019, 8:31 p.m., Na Li wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70512/ > ----------------------------------------------------------- > > (Updated May 2, 2019, 8:31 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath > Subramanian, and Xinran Tinney. > > > Repository: atlas > > > Description > ------- > > Impala generates lineage records for its commands. This new feature will read > Impala lineage file, convert the lineage record to Atlas entities and send > them to Atlas. In this way, Atlas can get lineage of Impala operation. > > The metadata referred in the lineage are captured in Hive Metastore hook and > sent to Atlas. This work is done in ATLAS-3148 > > This jira only supports the Impala command "create view". Following jira will > add support for more Impala commands. > > > Diffs > ----- > > addons/impala-bridge-shim/pom.xml PRE-CREATION > > addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > PRE-CREATION > addons/impala-bridge/pom.xml PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataTypes.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageQuery.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java > PRE-CREATION > addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-application.properties > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION > addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION > addons/impala-bridge/src/test/resources/users-credentials.properties > PRE-CREATION > pom.xml 7de5d31 > > > Diff: https://reviews.apache.org/r/70512/diff/13/ > > > Testing > ------- > > Run the tool in real cluster that has Atlas server with Impala lineage file > as input for creating view. The Atlas UI displays hive_lineage lineage and > hive_column_lineage. > Add new integration tests and they pass > > > Thanks, > > Na Li > >
