Issue with docker setup
Hello all, I am trying to use Apache atlas-docker setup to deploy a test framework? How ever it is not working for me. It worked perfectly fine a while back. 1) I was able to fo do unto step 5.1. I am not sure if 5.1 is needed before for 5.2 or not. Is it needed? 2) When I run 5.2, I get into errors. Am I missing some thing? las-hadoop.yml -f docker-compose.atlas-hbase.yml -f docker-compose.atlas-kafka.yml -f docker-compose.atlas-hive.yml up -d --remove-orphans atlas-db is up-to-date atlas-solr is up-to-date Starting atlas-docker_atlas-base_1 ... Starting atlas-docker_atlas-base_1 ... done Starting atlas-kafka ... done Starting atlas-hadoop ... done Starting atlas-hive... done ERROR: for atlas-hbase Container "3879a8299b65" is unhealthy. ERROR: Encountered errors while bringing up the project. Any suggestions? Regards, Sridhar K
Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance
> On Oct. 22, 2019, 6:31 p.m., Sridhar K wrote: > > addons/models/4000-MachineLearning/4010-ml_model.json > > Lines 123 (patched) > > <https://reviews.apache.org/r/71619/diff/7/?file=2169829#file2169829line123> > > > > I am not the right person for this comment. But, this name looks > > confusingis it a tag or image url? > > Na Li wrote: > Anand is very experienced in ML, and he said this name is commonly used > for that purpose. It contains where to get the container image of a model > > Anand Patil wrote: > "Image tag" is a standard Docker term: > https://docs.docker.com/engine/reference/commandline/tag/#tag-an-image-for-a-private-repository > It does include a sort of URL. I agree that the term is confusing, though. thanks for explaining. > On Oct. 22, 2019, 6:31 p.m., Sridhar K wrote: > > addons/models/4000-MachineLearning/4010-ml_model.json > > Lines 269 (patched) > > <https://reviews.apache.org/r/71619/diff/7/?file=2169829#file2169829line269> > > > > I am wondering if we should have a first class entity called "User" > > defined in base model and you have ml_user extend from it. what do you say? > > Na Li wrote: > what is parent for the entity "user"? > > Na Li wrote: > Made the changes > 1) define "AtlasUser" in base, derived from "Asset", it has "userName" > attribute > 2) define "ml_user", derived from "AtlasUser" and "DataSet". It only has > "metaData" attibute > > Anand Patil wrote: > If we are OK to inherit from Asset, should the other types (project, > build, deployment) inherit from Asset too? None of them are really datasets. > +Sridhar thoughts? I like this idea as wellI am assuming that they will have name, description and owner like the Asset has. - Sridhar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71619/#review218332 --- On Oct. 22, 2019, 8:04 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71619/ > --- > > (Updated Oct. 22, 2019, 8:04 p.m.) > > > Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, > Sridhar K, Madhan Neethiraj, and Sarath Subramanian. > > > Bugs: atlas-3464 > https://issues.apache.org/jira/browse/atlas-3464 > > > Repository: atlas > > > Description > --- > > Define entities used for Machine Learning Governance > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf > addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/71619/diff/8/ > > > Testing > --- > > verified it is valid json file > > > Thanks, > > Na Li > >
Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance
> On Oct. 22, 2019, 9:08 p.m., Sridhar K wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 203 (patched) > > <https://reviews.apache.org/r/71619/diff/8/?file=2169830#file2169830line203> > > > > Instead of modifying addons/models/-Area0/0010-base_model.json, can > > you add this change in addons/models/0000-Area0/0011-base_model.json > > Sridhar K wrote: > Sorryinstead of modifying > addons/models/-Area0/0010-base_model.json, can you add this change in > addons/models/-Area0/0012-base_model.json we want to add new changes model enhancements as new files with increasing names. - Sridhar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71619/#review218339 --- On Oct. 22, 2019, 8:04 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71619/ > --- > > (Updated Oct. 22, 2019, 8:04 p.m.) > > > Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, > Sridhar K, Madhan Neethiraj, and Sarath Subramanian. > > > Bugs: atlas-3464 > https://issues.apache.org/jira/browse/atlas-3464 > > > Repository: atlas > > > Description > --- > > Define entities used for Machine Learning Governance > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf > addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/71619/diff/8/ > > > Testing > --- > > verified it is valid json file > > > Thanks, > > Na Li > >
Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance
> On Oct. 22, 2019, 9:08 p.m., Sridhar K wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 203 (patched) > > <https://reviews.apache.org/r/71619/diff/8/?file=2169830#file2169830line203> > > > > Instead of modifying addons/models/-Area0/0010-base_model.json, can > > you add this change in addons/models/-Area0/0011-base_model.json Sorryinstead of modifying addons/models/-Area0/0010-base_model.json, can you add this change in addons/models/-Area0/0012-base_model.json - Sridhar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71619/#review218339 --- On Oct. 22, 2019, 8:04 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71619/ > --- > > (Updated Oct. 22, 2019, 8:04 p.m.) > > > Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, > Sridhar K, Madhan Neethiraj, and Sarath Subramanian. > > > Bugs: atlas-3464 > https://issues.apache.org/jira/browse/atlas-3464 > > > Repository: atlas > > > Description > --- > > Define entities used for Machine Learning Governance > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf > addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/71619/diff/8/ > > > Testing > --- > > verified it is valid json file > > > Thanks, > > Na Li > >
Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71619/#review218339 --- addons/models/-Area0/0010-base_model.json Lines 203 (patched) <https://reviews.apache.org/r/71619/#comment306009> Instead of modifying addons/models/-Area0/0010-base_model.json, can you add this change in addons/models/-Area0/0011-base_model.json - Sridhar K On Oct. 22, 2019, 8:04 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71619/ > --- > > (Updated Oct. 22, 2019, 8:04 p.m.) > > > Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, > Sridhar K, Madhan Neethiraj, and Sarath Subramanian. > > > Bugs: atlas-3464 > https://issues.apache.org/jira/browse/atlas-3464 > > > Repository: atlas > > > Description > --- > > Define entities used for Machine Learning Governance > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf > addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/71619/diff/8/ > > > Testing > --- > > verified it is valid json file > > > Thanks, > > Na Li > >
Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71619/#review218332 --- addons/models/4000-MachineLearning/4010-ml_model.json Lines 123 (patched) <https://reviews.apache.org/r/71619/#comment305998> I am not the right person for this comment. But, this name looks confusingis it a tag or image url? addons/models/4000-MachineLearning/4010-ml_model.json Lines 269 (patched) <https://reviews.apache.org/r/71619/#comment305999> I am wondering if we should have a first class entity called "User" defined in base model and you have ml_user extend from it. what do you say? - Sridhar K On Oct. 22, 2019, 6:17 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71619/ > --- > > (Updated Oct. 22, 2019, 6:17 p.m.) > > > Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, > Sridhar K, Madhan Neethiraj, and Sarath Subramanian. > > > Bugs: atlas-3464 > https://issues.apache.org/jira/browse/atlas-3464 > > > Repository: atlas > > > Description > --- > > Define entities used for Machine Learning Governance > > > Diffs > - > > addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/71619/diff/7/ > > > Testing > --- > > verified it is valid json file > > > Thanks, > > Na Li > >
Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71619/#review218328 --- Minor comments. all related to attribute names. Please submit with the updated review request. addons/models/4000-MachineLearning/4010-ml_model.json Lines 46 (patched) <https://reviews.apache.org/r/71619/#comment305991> is it different from Name? addons/models/4000-MachineLearning/4010-ml_model.json Lines 63 (patched) <https://reviews.apache.org/r/71619/#comment305992> Can we change it to createTime for consistency with other entities. addons/models/4000-MachineLearning/4010-ml_model.json Lines 71 (patched) <https://reviews.apache.org/r/71619/#comment305993> Can we please change it to modifiedTime for consistency with other entities. addons/models/4000-MachineLearning/4010-ml_model.json Lines 179 (patched) <https://reviews.apache.org/r/71619/#comment305994> Same comment as above. In general, we want to have consistent name for similar attributes even if they are from different entities. So this comment applies to all createdAt and updatedAt attributes. addons/models/4000-MachineLearning/4010-ml_model.json Lines 206 (patched) <https://reviews.apache.org/r/71619/#comment305995> Let us call this as deployedTime. If you have time based attributes have names ending with "*Time" for consistency. - Sridhar K On Oct. 22, 2019, 3:42 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71619/ > --- > > (Updated Oct. 22, 2019, 3:42 p.m.) > > > Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, > Sridhar K, Madhan Neethiraj, and Sarath Subramanian. > > > Bugs: atlas-3464 > https://issues.apache.org/jira/browse/atlas-3464 > > > Repository: atlas > > > Description > --- > > Define entities used for Machine Learning Governance > > > Diffs > - > > addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/71619/diff/6/ > > > Testing > --- > > verified it is valid json file > > > Thanks, > > Na Li > >
Re: Review Request 71500: ATLAS-3414: Include metastore catalog name in hive database name and qualifiedName
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71500/#review217824 --- What is the plan on keeping catalog name as an attribute of Database Entitiy. Is there a reason why we did not do this in this version? - Sridhar K On Sept. 18, 2019, 5:41 a.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71500/ > --- > > (Updated Sept. 18, 2019, 5:41 a.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > and Madhan Neethiraj. > > > Bugs: ATLAS_3414 > https://issues.apache.org/jira/browse/ATLAS_3414 > > > Repository: atlas > > > Description > --- > > It is possible to create new catalogs in hive metastore. For non-default > metastore catalogs, we need to include the catalogName in hive database name > and qualifiedName to differentiate between default 'hive' catalog database > entities > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 2f1bb0eae > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java > 0eee7c143 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > c6c0c663c > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java > f4f933ee1 > > > Diff: https://reviews.apache.org/r/71500/diff/1/ > > > Testing > --- > > Precommit: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1416/console > > > Thanks, > > Sarath Subramanian > >
Re: Review Request 71470: ATLAS-3378: Update JanusGraph to 0.4.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71470/#review217691 --- Ship it! Sarath looks good to me. You probably did it. But,I am wondering if have any simple queries done with functionality that use rinkerpop. - Sridhar K On Sept. 11, 2019, 7:20 p.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71470/ > --- > > (Updated Sept. 11, 2019, 7:20 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Nixon Rodrigues. > > > Bugs: ATLAS-3378 > https://issues.apache.org/jira/browse/ATLAS-3378 > > > Repository: atlas > > > Description > --- > > Update JanusGraph to 0.4.0 > > > Diffs > - > > graphdb/api/pom.xml 9cdbe6650 > graphdb/janus/pom.xml adc9a9e16 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > 4d581ec95 > pom.xml c18836b20 > > > Diff: https://reviews.apache.org/r/71470/diff/1/ > > > Testing > --- > > Precommit: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1401/console > > Validated that Atlas with Janusgraph 0.4.0 is backward compatible with > earlier janusgraph version (0.3.1) > > > Thanks, > > Sarath Subramanian > >
Re: Review Request 71319: ATLAS-3380: Regression: After moving a table to a different database , db attribute in the table entity still points to the old db.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71319/#review217303 --- Ship it! Ship It! - Sridhar K On Aug. 19, 2019, 9:58 p.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71319/ > --- > > (Updated Aug. 19, 2019, 9:58 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Nixon Rodrigues. > > > Bugs: ATLAS-3380 > https://issues.apache.org/jira/browse/ATLAS-3380 > > > Repository: atlas > > > Description > --- > > Hive commands: > > 1.create database db1; > > 2.create table table1(id int); > > 3.alter table table1 rename to db1.table2; > > > > In table2's entity definition , db attribute is still pointing to default db > , expected is db1. Other attributes like qualifiedName , sd's qualifiedName > are pointing to db1 correctly. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java > f496b788d > intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > 5b24ef14e > > > Diff: https://reviews.apache.org/r/71319/diff/1/ > > > Testing > --- > > Manually validated the fix works. > > > Thanks, > > Sarath Subramanian > >
Re: Review Request 71294: ATLAS-3374: fix for incorrect basic/quick search results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71294/#review217220 --- Ship it! Ship It! - Sridhar K On Aug. 15, 2019, 9:48 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71294/ > --- > > (Updated Aug. 15, 2019, 9:48 a.m.) > > > Review request for atlas, Ashutosh Mestry, keval bhatt, Sridhar K, Nixon > Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3374 > https://issues.apache.org/jira/browse/ATLAS-3374 > > > Repository: atlas > > > Description > --- > > - after an update to typesystem, the new type-registry wasn't initialized > with index-field-names for existing attributes, resulting in incorrect > population of search handler i.e. had an null entry > - insufficient error handling cased the search handler to be populated with > incomplate index-field names, which caused incorrect search > > > Diffs > - > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphIndexClient.java > 113ea6c57 > intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 5c94c338f > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 8d8e4ad92 > > repository/src/main/java/org/apache/atlas/repository/graph/SolrIndexHelper.java > f337fb359 > > > Diff: https://reviews.apache.org/r/71294/diff/1/ > > > Testing > --- > > - verified that the search returns expected results for the failure cases > given in the JIRA > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1356/ > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 71089: ATLAS-3338: avoid repeated calls to get index-field name during startup
> On July 18, 2019, 8:10 a.m., Sridhar K wrote: > > Ship It! Thanks for fixing it. - Sridhar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71089/#review216713 --- On July 17, 2019, 6:45 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71089/ > --- > > (Updated July 17, 2019, 6:45 a.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, > Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Nikhil Bonte, Nixon > Rodrigues, Saqeeb Shaikh, and Sarath Subramanian. > > > Bugs: ATLAS-3338 > https://issues.apache.org/jira/browse/ATLAS-3338 > > > Repository: atlas > > > Description > --- > > updated resolveIndexFieldName() to check the attribute instance in super-type > i.e. the type where the attribute is defined-in; only when this instance > doesn't have index-field name set, graph API is called to retrieve the value > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 95ae5049e > > > Diff: https://reviews.apache.org/r/71089/diff/2/ > > > Testing > --- > > - manaully verified that only one call is made to graph API during > type-system initialization/update > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1271 > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 71089: ATLAS-3338: avoid repeated calls to get index-field name during startup
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71089/#review216713 --- Ship it! Ship It! - Sridhar K On July 17, 2019, 6:45 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71089/ > --- > > (Updated July 17, 2019, 6:45 a.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, > Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Nikhil Bonte, Nixon > Rodrigues, Saqeeb Shaikh, and Sarath Subramanian. > > > Bugs: ATLAS-3338 > https://issues.apache.org/jira/browse/ATLAS-3338 > > > Repository: atlas > > > Description > --- > > updated resolveIndexFieldName() to check the attribute instance in super-type > i.e. the type where the attribute is defined-in; only when this instance > doesn't have index-field name set, graph API is called to retrieve the value > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 95ae5049e > > > Diff: https://reviews.apache.org/r/71089/diff/2/ > > > Testing > --- > > - manaully verified that only one call is made to graph API during > type-system initialization/update > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1271 > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace
> On July 10, 2019, 10:12 p.m., Madhan Neethiraj wrote: > > Ship It! I am not convinved that this is correct approach. - Sridhar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71042/#review216503 --- On July 10, 2019, 8:11 p.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71042/ > --- > > (Updated July 10, 2019, 8:11 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Nixon Rodrigues. > > > Bugs: ATLAS-3321 > https://issues.apache.org/jira/browse/ATLAS-3321 > > > Repository: atlas > > > Description > --- > > Atlas has the notion of clusterName which is used as a global namespace in > which you can define the context in which entity instances are defined and > grouped. A cluster name defines a scope of operation in atlas and is > specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), > fs_path (File System), storm_topology (Storm), spark_db (Spark) > > cluster name is also used to construct qualifiedName attribute in atlas - > > hive_db - db_name@cluster_name > hive_table - db_name.table_name@cluster_name > hive_column - db_name.table_name.column_name@cluster_name > Cluster name works for simple environments wherein we have a single cluster > and single service instances running (single Hive or HBase service). In > complex deployments - a single cluster may host multiple instances of the > same service (two Hive services - Hive-1 and Hive-2), for such cases cluster > name doesn't help to provide context to individual service instances and > metadata groupings. > > So we introduce - "metadata namespace" where a single cluster can contain > multiple metadata namespaces. The qualified name for atlas will be generated > using metadata namespace name instead of cluster name. > > For each hook enabled we specify the property - "atlas.metadata.namespace" > which defines the scope/context of the metadata. > > > Diffs > - > > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java > 1825cd290 > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java > 1765c1848 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 5f8f846ae > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java > 76d6fe612 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > ffa56cea0 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > b3663da34 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java > cbee7bf14 > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java > d55aa5370 > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java > 1305f65a6 > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java > b5fdb6dbd > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java > 4ea484f52 > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > f1d02375c > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > 8ebb385cf > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > a7b9b0cd0 > > addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java > 8755c9e60 > > addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java > 5397a4b0b > > addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java > 97668a3c6 > common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d > notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 0030276c6 > > > Diff: https://reviews.apache.org/r/71042/diff/2/ > > > Testing > --- > > Testing in progress > > > Thanks, > > Sarath Subramanian > >
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace
> On July 10, 2019, 5:55 a.m., Sridhar K wrote: > > Hello Sarath, > > Is this some thing new that we are doing? I understand the reasoning. I > > don't see a need for configuration "atlas.metadata.namespace". I suggest > > that we just use the . format. It is hard to > > manage with configurations like this. What would happen if this > > configuration is changed? What will you do with existing objects when this > > value is changed. This needs some kind of design exercise. Can we please > > have a . meeting for it. > > Madhan Neethiraj wrote: > Sridhar - current use of 'clusterName' works when all services that > access (CRUD) a resource run in the same cluster. There are cases where > services running in a cluster that is different from where the resource > resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services > running in both clusters can access Hive metastore running in "cl1". In such > scenarios holding on to 'clusterName' would be confusing; hence the move to a > generic name 'metadata.namespace'. It makes it intutive to ask the services > set this configuration, instead of confusing them with 'clusterName'. Hope > this helps. > > Sridhar K wrote: > I see your point. But, the issue is in the possibility of wrongly > configured names and flexibility of changing the names. In the example you > gave, I would expect HMS to have the convention of "cl1.hms1" as auto > generated configuration and Hive-Server resources like hive_processes should > have their name spaces as "cl1.hive-server-1" and " cl2.hive-server-1". I > think that this gives a general understanding on namespace hierarchy. If it > is user configurable, user might accidentally give the same name for every > cluster resulting in mixups. So, I think that we need to give some thinking. > You might have given some thinking on it. If so, I would like to understand > it. If we allow the user to give values for namespace, we might have to have this information captured as first class Atlas Entity as well--for auditing, goveranance point of view. - Sridhar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71042/#review216470 --- On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71042/ > --- > > (Updated July 9, 2019, 11:53 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Nixon Rodrigues. > > > Bugs: ATLAS-3321 > https://issues.apache.org/jira/browse/ATLAS-3321 > > > Repository: atlas > > > Description > --- > > Atlas has the notion of clusterName which is used as a global namespace in > which you can define the context in which entity instances are defined and > grouped. A cluster name defines a scope of operation in atlas and is > specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), > fs_path (File System), storm_topology (Storm), spark_db (Spark) > > cluster name is also used to construct qualifiedName attribute in atlas - > > hive_db - db_name@cluster_name > hive_table - db_name.table_name@cluster_name > hive_column - db_name.table_name.column_name@cluster_name > Cluster name works for simple environments wherein we have a single cluster > and single service instances running (single Hive or HBase service). In > complex deployments - a single cluster may host multiple instances of the > same service (two Hive services - Hive-1 and Hive-2), for such cases cluster > name doesn't help to provide context to individual service instances and > metadata groupings. > > So we introduce - "metadata namespace" where a single cluster can contain > multiple metadata namespaces. The qualified name for atlas will be generated > using metadata namespace name instead of cluster name. > > For each hook enabled we specify the property - "atlas.metadata.namespace" > which defines the scope/context of the metadata. > > > Diffs > - > > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java > 1825cd290 > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java > 1765c1848 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 5f
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace
> On July 10, 2019, 5:55 a.m., Sridhar K wrote: > > Hello Sarath, > > Is this some thing new that we are doing? I understand the reasoning. I > > don't see a need for configuration "atlas.metadata.namespace". I suggest > > that we just use the . format. It is hard to > > manage with configurations like this. What would happen if this > > configuration is changed? What will you do with existing objects when this > > value is changed. This needs some kind of design exercise. Can we please > > have a . meeting for it. > > Madhan Neethiraj wrote: > Sridhar - current use of 'clusterName' works when all services that > access (CRUD) a resource run in the same cluster. There are cases where > services running in a cluster that is different from where the resource > resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services > running in both clusters can access Hive metastore running in "cl1". In such > scenarios holding on to 'clusterName' would be confusing; hence the move to a > generic name 'metadata.namespace'. It makes it intutive to ask the services > set this configuration, instead of confusing them with 'clusterName'. Hope > this helps. I see your point. But, the issue is in the possibility of wrongly configured names and flexibility of changing the names. In the example you gave, I would expect HMS to have the convention of "cl1.hms1" as auto generated configuration and Hive-Server resources like hive_processes should have their name spaces as "cl1.hive-server-1" and " cl2.hive-server-1". I think that this gives a general understanding on namespace hierarchy. If it is user configurable, user might accidentally give the same name for every cluster resulting in mixups. So, I think that we need to give some thinking. You might have given some thinking on it. If so, I would like to understand it. - Sridhar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71042/#review216470 --- On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71042/ > ----------- > > (Updated July 9, 2019, 11:53 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Nixon Rodrigues. > > > Bugs: ATLAS-3321 > https://issues.apache.org/jira/browse/ATLAS-3321 > > > Repository: atlas > > > Description > --- > > Atlas has the notion of clusterName which is used as a global namespace in > which you can define the context in which entity instances are defined and > grouped. A cluster name defines a scope of operation in atlas and is > specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), > fs_path (File System), storm_topology (Storm), spark_db (Spark) > > cluster name is also used to construct qualifiedName attribute in atlas - > > hive_db - db_name@cluster_name > hive_table - db_name.table_name@cluster_name > hive_column - db_name.table_name.column_name@cluster_name > Cluster name works for simple environments wherein we have a single cluster > and single service instances running (single Hive or HBase service). In > complex deployments - a single cluster may host multiple instances of the > same service (two Hive services - Hive-1 and Hive-2), for such cases cluster > name doesn't help to provide context to individual service instances and > metadata groupings. > > So we introduce - "metadata namespace" where a single cluster can contain > multiple metadata namespaces. The qualified name for atlas will be generated > using metadata namespace name instead of cluster name. > > For each hook enabled we specify the property - "atlas.metadata.namespace" > which defines the scope/context of the metadata. > > > Diffs > - > > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java > 1825cd290 > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java > 1765c1848 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 5f8f846ae > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java > 76d6fe612 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > ffa56cea0 > > addons/hive-b
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71042/#review216470 --- Hello Sarath, Is this some thing new that we are doing? I understand the reasoning. I don't see a need for configuration "atlas.metadata.namespace". I suggest that we just use the . format. It is hard to manage with configurations like this. What would happen if this configuration is changed? What will you do with existing objects when this value is changed. This needs some kind of design exercise. Can we please have a . meeting for it. - Sridhar K On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71042/ > --- > > (Updated July 9, 2019, 11:53 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Nixon Rodrigues. > > > Bugs: ATLAS-3321 > https://issues.apache.org/jira/browse/ATLAS-3321 > > > Repository: atlas > > > Description > --- > > Atlas has the notion of clusterName which is used as a global namespace in > which you can define the context in which entity instances are defined and > grouped. A cluster name defines a scope of operation in atlas and is > specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), > fs_path (File System), storm_topology (Storm), spark_db (Spark) > > cluster name is also used to construct qualifiedName attribute in atlas - > > hive_db - db_name@cluster_name > hive_table - db_name.table_name@cluster_name > hive_column - db_name.table_name.column_name@cluster_name > Cluster name works for simple environments wherein we have a single cluster > and single service instances running (single Hive or HBase service). In > complex deployments - a single cluster may host multiple instances of the > same service (two Hive services - Hive-1 and Hive-2), for such cases cluster > name doesn't help to provide context to individual service instances and > metadata groupings. > > So we introduce - "metadata namespace" where a single cluster can contain > multiple metadata namespaces. The qualified name for atlas will be generated > using metadata namespace name instead of cluster name. > > For each hook enabled we specify the property - "atlas.metadata.namespace" > which defines the scope/context of the metadata. > > > Diffs > - > > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java > 1825cd290 > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java > 1765c1848 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 5f8f846ae > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java > 76d6fe612 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > ffa56cea0 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > 05d79df29 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java > cbee7bf14 > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java > d55aa5370 > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java > 1305f65a6 > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java > b5fdb6dbd > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java > 4ea484f52 > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > f1d02375c > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > 8ebb385cf > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > a7b9b0cd0 > > addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java > 8755c9e60 > > addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java > 5397a4b0b > > addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java > 97668a3c6 > common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d > > > Diff: https://reviews.apache.org/r/71042/diff/1/ > > > Testing > --- > > Testing in progress > > > Thanks, > > Sarath Subramanian > >
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71006/#review216439 --- - Sridhar K On July 8, 2019, 1:26 p.m., Saqeeb Shaikh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71006/ > --- > > (Updated July 8, 2019, 1:26 p.m.) > > > Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon > Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3313 > https://issues.apache.org/jira/browse/ATLAS-3313 > > > Repository: atlas > > > Description > --- > > Currently Atlas server processes notifications from Kafka topic named > ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). > The name of the Kafka topic to which an Atlas Hook publishes notifications > can be configured via configuration atlas.notification.hook.topic.name. > However, since Atlas server processes notifications from a single topic, all > hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic. > > To enable different hooks to publish to different Kafka topics, Atlas server > should support processing notifications from multiple Kafka topics. > > > Add following properties in atlas: > atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK > (comma separated without space) > atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 > intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > 5c840c3 > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java > b04aba9 > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 449eb6f > notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java > 847caa3 > > notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java > 263903b > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java > c963830 > repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java > d5a4412 > repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java > f658caa > repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java > b56019e > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > 1f8e810 > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java > fb3ff26 > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java > 3e35511 > > > Diff: https://reviews.apache.org/r/71006/diff/2/ > > > Testing > --- > > Tested by publishing create hive table JSON using kafka-console-producer on > custom topics configured above. I was able to search the entities in atlas. > Also tested basic use case by creating database and table from hive console. > > mvn clean package > Some test cases are failing on my local (will work on fixing the same). > > > Thanks, > > Saqeeb Shaikh > >
Re: Review Request 70930: ATLAS-3297: cache added/update entities in request-context, for use in entity-change listeners
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70930/#review216200 --- Ship it! Ship It! - Sridhar K On June 22, 2019, 6:57 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70930/ > --- > > (Updated June 22, 2019, 6:57 p.m.) > > > Review request for atlas, Ayub Pathan, Ashutosh Mestry, Binit Gutka, Aadarsh > Jajodia, keval bhatt, Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Mehul > Parikh, Nikhil Bonte, Nixon Rodrigues, Saqeeb Shaikh, Sarath Subramanian, and > Sharmadha Sainath. > > > Bugs: ATLAS-3297 > https://issues.apache.org/jira/browse/ATLAS-3297 > > > Repository: atlas > > > Description > --- > > Update EntityGraphMapper to cache entities after performing create/update > operation. With these entities in cach, entity-change listeners avoid reading > from store. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 64f5503fe > > > Diff: https://reviews.apache.org/r/70930/diff/1/ > > > Testing > --- > > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1219/ > - processed a large number of notifications before and after this update - > observed reduction of 33% time (19h vs 12h15m) > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 70840: ATLAS-3276: Fix stale transactions in atlas due to ATLAS-3246 (Free-text search)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70840/#review215902 --- Ship it! Ship It! - Sridhar K On June 13, 2019, 9:08 p.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70840/ > --- > > (Updated June 13, 2019, 9:08 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > and Madhan Neethiraj. > > > Bugs: ATLAS-3276 > https://issues.apache.org/jira/browse/ATLAS-3276 > > > Repository: atlas > > > Description > --- > > Fix stale transactions in atlas due to ATLAS-3246 > > > Diffs > - > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphIndexClient.java > 4dd641da7 > repository/src/main/java/org/apache/atlas/services/MetricsService.java > 8fb68e926 > repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java > 0c86aff2c > > > Diff: https://reviews.apache.org/r/70840/diff/5/ > > > Testing > --- > > No longer see stale transaction alerts in atlas logs. > > Precommit: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1199/console > > > Thanks, > > Sarath Subramanian > >
Re: Review Request 70840: ATLAS-3276: Fix stale transactions in atlas due to ATLAS-3246 (Free-text search)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70840/#review215862 --- repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java Line 167 (original), 167 (patched) <https://reviews.apache.org/r/70840/#comment302804> The API does not match the normal tx api normally, you would have open tx, do some action, commit or rollback tx. I don't see open in the codeAm I missing some thing? - Sridhar K On June 12, 2019, 9:04 p.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70840/ > --- > > (Updated June 12, 2019, 9:04 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > and Madhan Neethiraj. > > > Bugs: ATLAS-3276 > https://issues.apache.org/jira/browse/ATLAS-3276 > > > Repository: atlas > > > Description > --- > > Fix stale transactions in atlas due to ATLAS-3246 > > > Diffs > - > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphIndexClient.java > 4dd641da7 > repository/src/main/java/org/apache/atlas/services/MetricsService.java > 8fb68e926 > repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java > 0c86aff2c > > > Diff: https://reviews.apache.org/r/70840/diff/3/ > > > Testing > --- > > No longer see stale transaction alerts in atlas logs. > > Precommit: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1199/console > > > Thanks, > > Sarath Subramanian > >
Re: Review Request 70789: ATLAS-3262: disabled full-text index population when free-text is enabled i.e. for Solr index store
> On June 7, 2019, 1:55 p.m., Sridhar K wrote: > > intg/src/main/java/org/apache/atlas/ApplicationProperties.java > > Lines 52 (patched) > > <https://reviews.apache.org/r/70789/diff/2-3/?file=2147960#file2147960line52> > > > > It is confusing herewhy two properties? > > Madhan Neethiraj wrote: > atlas.search.fulltext.enable was an exising property, which enables > disabling of fulltext-search feature. > > atlas.search.freetext.enable was introduced in this commit, to support > disabling of freetext feature; primarily to address IT failures caused by > used of embedded-solr. I don't think it is right to introduce a variable to fix INT test. Let me investigate this little bit more. - Sridhar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70789/#review215745 --- On June 7, 2019, 8 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70789/ > --- > > (Updated June 7, 2019, 8 a.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, > Sridhar K, Le Ma, Nikhil Bonte, Nixon Rodrigues, Saqeeb Shaikh, and Sarath > Subramanian. > > > Bugs: ATLAS-3262 > https://issues.apache.org/jira/browse/ATLAS-3262 > > > Repository: atlas > > > Description > --- > > - skipped full-text index computation & population when free-text is enabled > - added FreeTextRequestHandlerPatch to create /freetext request handler in > Solr > - fixed free-text search handling of typeName filter > > > Diffs > - > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > edab08cc0 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java > 18e6976a4 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphSolrIndexClient.java > a55fc3689 > graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java > 0d5ae4f84 > intg/src/main/java/org/apache/atlas/ApplicationProperties.java 6d0153f7d > intg/src/main/java/org/apache/atlas/listener/ChangedTypeDefs.java 0636677ae > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 1cd507c3b > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > c4b5ffb12 > > repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java > b1a84c742 > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 50edb9faf > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > 2e09beda0 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 56655a845 > > repository/src/main/java/org/apache/atlas/repository/graph/IndexChangeListener.java > 9cde947aa > > repository/src/main/java/org/apache/atlas/repository/graph/SolrIndexHelper.java > 20a517ff1 > > repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java > 259f24674 > > repository/src/main/java/org/apache/atlas/repository/patches/FreeTextRequestHandlerPatch.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 035b02ce4 > > repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java > 1d296057b > repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java > 38de885cf > > > Diff: https://reviews.apache.org/r/70789/diff/3/ > > > Testing > --- > > - verified that free-text handler is created by the patch for the first time > this enhancement is applied in an existing env > - verified that the free-text search handles typeName filter correctly > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1176/ > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 70789: ATLAS-3262: disabled full-text index population when free-text is enabled i.e. for Solr index store
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70789/#review215745 --- Review is not yet complete. intg/src/main/java/org/apache/atlas/ApplicationProperties.java Lines 52 (patched) <https://reviews.apache.org/r/70789/#comment302556> It is confusing herewhy two properties? - Sridhar K On June 7, 2019, 8 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70789/ > --- > > (Updated June 7, 2019, 8 a.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, > Sridhar K, Le Ma, Nikhil Bonte, Nixon Rodrigues, Saqeeb Shaikh, and Sarath > Subramanian. > > > Bugs: ATLAS-3262 > https://issues.apache.org/jira/browse/ATLAS-3262 > > > Repository: atlas > > > Description > --- > > - skipped full-text index computation & population when free-text is enabled > - added FreeTextRequestHandlerPatch to create /freetext request handler in > Solr > - fixed free-text search handling of typeName filter > > > Diffs > - > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > edab08cc0 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java > 18e6976a4 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphSolrIndexClient.java > a55fc3689 > graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java > 0d5ae4f84 > intg/src/main/java/org/apache/atlas/ApplicationProperties.java 6d0153f7d > intg/src/main/java/org/apache/atlas/listener/ChangedTypeDefs.java 0636677ae > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 1cd507c3b > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > c4b5ffb12 > > repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java > b1a84c742 > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 50edb9faf > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > 2e09beda0 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 56655a845 > > repository/src/main/java/org/apache/atlas/repository/graph/IndexChangeListener.java > 9cde947aa > > repository/src/main/java/org/apache/atlas/repository/graph/SolrIndexHelper.java > 20a517ff1 > > repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java > 259f24674 > > repository/src/main/java/org/apache/atlas/repository/patches/FreeTextRequestHandlerPatch.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 035b02ce4 > > repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java > 1d296057b > repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java > 38de885cf > > > Diff: https://reviews.apache.org/r/70789/diff/3/ > > > Testing > --- > > - verified that free-text handler is created by the patch for the first time > this enhancement is applied in an existing env > - verified that the free-text search handles typeName filter correctly > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1176/ > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 70789: ATLAS-3262: disabled full-text index population when free-text is enabled i.e. for Solr index store
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70789/#review215726 --- Ship it! Ship It! - Sridhar K On June 6, 2019, 8:18 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70789/ > --- > > (Updated June 6, 2019, 8:18 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, > Sridhar K, Le Ma, Nikhil Bonte, Nixon Rodrigues, Saqeeb Shaikh, and Sarath > Subramanian. > > > Bugs: ATLAS-3262 > https://issues.apache.org/jira/browse/ATLAS-3262 > > > Repository: atlas > > > Description > --- > > - skipped full-text index computation & population when free-text is enabled > - added FreeTextRequestHandlerPatch to create /freetext request handler in > Solr > - fixed free-text search handling of typeName filter > > > Diffs > - > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > edab08cc0 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphSolrIndexClient.java > a55fc3689 > intg/src/main/java/org/apache/atlas/ApplicationProperties.java 6d0153f7d > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 1cd507c3b > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > c4b5ffb12 > > repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java > b1a84c742 > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 50edb9faf > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > 2e09beda0 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 56655a845 > > repository/src/main/java/org/apache/atlas/repository/graph/SolrIndexHelper.java > 20a517ff1 > > repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java > 259f24674 > > repository/src/main/java/org/apache/atlas/repository/patches/FreeTextRequestHandlerPatch.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 035b02ce4 > > repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java > 1d296057b > repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java > 38de885cf > > > Diff: https://reviews.apache.org/r/70789/diff/2/ > > > Testing > --- > > - verified that free-text handler is created by the patch for the first time > this enhancement is applied in an existing env > - verified that the free-text search handles typeName filter correctly > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1176/ > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 70789: ATLAS-3262: disabled full-text index population when free-text is enabled i.e. for Solr index store
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70789/#review215724 --- Ship it! Ship It! - Sridhar K On June 6, 2019, 8:18 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70789/ > --- > > (Updated June 6, 2019, 8:18 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, > Sridhar K, Le Ma, Nikhil Bonte, Nixon Rodrigues, Saqeeb Shaikh, and Sarath > Subramanian. > > > Bugs: ATLAS-3262 > https://issues.apache.org/jira/browse/ATLAS-3262 > > > Repository: atlas > > > Description > --- > > - skipped full-text index computation & population when free-text is enabled > - added FreeTextRequestHandlerPatch to create /freetext request handler in > Solr > - fixed free-text search handling of typeName filter > > > Diffs > - > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > edab08cc0 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphSolrIndexClient.java > a55fc3689 > intg/src/main/java/org/apache/atlas/ApplicationProperties.java 6d0153f7d > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 1cd507c3b > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > c4b5ffb12 > > repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java > b1a84c742 > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 50edb9faf > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > 2e09beda0 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 56655a845 > > repository/src/main/java/org/apache/atlas/repository/graph/SolrIndexHelper.java > 20a517ff1 > > repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java > 259f24674 > > repository/src/main/java/org/apache/atlas/repository/patches/FreeTextRequestHandlerPatch.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 035b02ce4 > > repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java > 1d296057b > repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java > 38de885cf > > > Diff: https://reviews.apache.org/r/70789/diff/2/ > > > Testing > --- > > - verified that free-text handler is created by the patch for the first time > this enhancement is applied in an existing env > - verified that the free-text search handles typeName filter correctly > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1176/ > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 70789: ATLAS-3262: disabled full-text index population when free-text is enabled i.e. for Solr index store
> On June 6, 2019, 4:50 p.m., Sridhar K wrote: > > repository/src/main/java/org/apache/atlas/repository/patches/FreeTextRequestHandlerPatch.java > > Lines 27 (patched) > > <https://reviews.apache.org/r/70789/diff/1/?file=2147935#file2147935line27> > > > > Can we remove this class initialize the SolrIndexHelper in PatchManager > > instead. I am not sure, if we want this to be created in a patch setting. > > > > I rather have it done explicitly AtlasPatchManager after going through > > all patches in the system. Otherwise, the new patches might be added after > > this patch and we might some def types or patches--may be listerner in > > GraphBackedIndexHelper will cover this scenario. But, I still think that > > this should be initialized by PatchManager after all patches are run as an > > explicit step. > > Madhan Neethiraj wrote: > This patch needs to be applied only once - when Atlas with this commit is > run on an earlier deployment. Else, the request-handler will not be created > until a typedef change is performed (like CRUD of entity-typedef, > classification-typedef). Oh yes. That is true. - Sridhar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70789/#review215718 --- On June 6, 2019, 8:18 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70789/ > --- > > (Updated June 6, 2019, 8:18 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, > Sridhar K, Le Ma, Nikhil Bonte, Nixon Rodrigues, Saqeeb Shaikh, and Sarath > Subramanian. > > > Bugs: ATLAS-3262 > https://issues.apache.org/jira/browse/ATLAS-3262 > > > Repository: atlas > > > Description > --- > > - skipped full-text index computation & population when free-text is enabled > - added FreeTextRequestHandlerPatch to create /freetext request handler in > Solr > - fixed free-text search handling of typeName filter > > > Diffs > - > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > edab08cc0 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphSolrIndexClient.java > a55fc3689 > intg/src/main/java/org/apache/atlas/ApplicationProperties.java 6d0153f7d > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 1cd507c3b > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > c4b5ffb12 > > repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java > b1a84c742 > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 50edb9faf > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > 2e09beda0 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 56655a845 > > repository/src/main/java/org/apache/atlas/repository/graph/SolrIndexHelper.java > 20a517ff1 > > repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java > 259f24674 > > repository/src/main/java/org/apache/atlas/repository/patches/FreeTextRequestHandlerPatch.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 035b02ce4 > > repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java > 1d296057b > repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java > 38de885cf > > > Diff: https://reviews.apache.org/r/70789/diff/2/ > > > Testing > --- > > - verified that free-text handler is created by the patch for the first time > this enhancement is applied in an existing env > - verified that the free-text search handles typeName filter correctly > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1176/ > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 70789: ATLAS-3262: disabled full-text index population when free-text is enabled i.e. for Solr index store
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70789/#review215718 --- Thanks for doing this so quickly. I do have some suggestions. thanks, Sridhar graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphSolrIndexClient.java Line 38 (original), 38 (patched) <https://reviews.apache.org/r/70789/#comment302521> I liked original name. If any one in community wants implement other AtlasGZraphIndexClients, they can add their own. graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphSolrIndexClient.java Lines 52 (patched) <https://reviews.apache.org/r/70789/#comment302524> I like it. graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphSolrIndexClient.java Line 63 (original), 61 (patched) <https://reviews.apache.org/r/70789/#comment302523> Thanks for noticing this. repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java Line 53 (original) <https://reviews.apache.org/r/70789/#comment302522> I am not sure why you removed it. Ofcourse, it is an optimization. repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java Line 95 (original), 70 (patched) <https://reviews.apache.org/r/70789/#comment302525> thx. repository/src/main/java/org/apache/atlas/discovery/SearchContext.java Line 79 (original), 77 (patched) <https://reviews.apache.org/r/70789/#comment302526> Never felt good with static block. Thanks for finding a better place for this. repository/src/main/java/org/apache/atlas/discovery/SearchContext.java Lines 124 (patched) <https://reviews.apache.org/r/70789/#comment302527> I prefer to not have so many lines. If funcional separation is needed, can we remove one line here? repository/src/main/java/org/apache/atlas/repository/graph/SolrIndexHelper.java Line 109 (original), 113 (patched) <https://reviews.apache.org/r/70789/#comment302528> I know what you wanted to do here. But, this is user entered value, I would prefer it to be throwing an exception early on so that the developers can notice it and fix it. Otherwise, they might not realize their mistake if eat up the issue silently without throwing exception. repository/src/main/java/org/apache/atlas/repository/patches/FreeTextRequestHandlerPatch.java Lines 27 (patched) <https://reviews.apache.org/r/70789/#comment302530> Can we remove this class initialize the SolrIndexHelper in PatchManager instead. I am not sure, if we want this to be created in a patch setting. I rather have it done explicitly AtlasPatchManager after going through all patches in the system. Otherwise, the new patches might be added after this patch and we might some def types or patches--may be listerner in GraphBackedIndexHelper will cover this scenario. But, I still think that this should be initialized by PatchManager after all patches are run as an explicit step. repository/src/main/java/org/apache/atlas/repository/patches/FreeTextRequestHandlerPatch.java Lines 31 (patched) <https://reviews.apache.org/r/70789/#comment302529> searche ==> searches. - Sridhar K On June 6, 2019, 7:41 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70789/ > --- > > (Updated June 6, 2019, 7:41 a.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, > Sridhar K, Le Ma, Nikhil Bonte, Nixon Rodrigues, Saqeeb Shaikh, and Sarath > Subramanian. > > > Bugs: ATLAS-3262 > https://issues.apache.org/jira/browse/ATLAS-3262 > > > Repository: atlas > > > Description > --- > > - skipped full-text index computation & population when free-text is enabled > - added FreeTextRequestHandlerPatch to create /freetext request handler in > Solr > - fixed free-text search handling of typeName filter > > > Diffs > - > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > edab08cc0 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphSolrIndexClient.java > a55fc3689 > intg/src/main/java/org/apache/atlas/ApplicationProperties.java 6d0153f7d > > repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java > b1a84c742 > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 50edb9faf > > repository/src/main/java/org/apache/atlas/repository/graph/Grap
Re: Review Request 70606: ATLAS-2901: Change classification name rule - Support multiple languages (unicode) and do not allow dot mark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70606/#review215605 --- Ship it! Ship It! - Sridhar K On May 21, 2019, 12:06 a.m., Yu-Hsin Shih wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70606/ > --- > > (Updated May 21, 2019, 12:06 a.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-2901 > https://issues.apache.org/jira/browse/ATLAS-2901 > > > Repository: atlas > > > Description > --- > > 1. Allow Unicode in classification name and follow the original regex rule as > possible as we can. > 2. Update the error message to reflect that the rule allows space. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java c04f561c1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java > dd020d183 > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2Test.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/70606/diff/2/ > > > Testing > --- > > Manual tested by using Chinese characters in both Classification and its > attribute. > Add unit test cases. > > > File Attachments > > > Classification def > > https://reviews.apache.org/media/uploaded/files/2019/05/07/f629f9db-e858-4f11-a018-5d91aba0a286__Screen_Shot_2019-05-07_at_1.50.01_PM.png > Entity Association > > https://reviews.apache.org/media/uploaded/files/2019/05/07/d36894df-5acf-4227-b591-11029bf3f009__Screen_Shot_2019-05-07_at_1.49.44_PM.png > > > Thanks, > > Yu-Hsin Shih > >
Re: Review Request 70701: ATLAS-3221: Refactor Impala integration code
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70701/#review215480 --- Ship it! Ship It! - Sridhar K On May 22, 2019, 8:54 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70701/ > --- > > (Updated May 22, 2019, 8:54 p.m.) > > > Review request for atlas and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > The code under impala-bridge module is under the folder > "addons/impala-bridge/src/main/java/org.apache.atlas.impala/", they should be > put under the folder > "addons/impala-bridge/src/main/java/org/apache/atlas/impala/" to follow the > convention. > > > Diffs > - > > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > 7c9abc8 > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java > 88faace > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java > 2afb4eb > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java > b9d6cbb > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java > 232a569 > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java > b9dd894 > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > 63c5f87 > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > 0dc520c > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java > 10ce448 > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java > 892ee9b > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java > a3ddf53 > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java > a893b88 > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java > 27bdc72 > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java > 8ec3f85 > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java > 251507e > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java > 0a664fc > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java > 0bd236d > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/ImpalaLineageTool.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaIdentifierParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDependencyType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaNode.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaOperationType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaQuery.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaVertexType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageEdge.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertex.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertexMetadata.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/70701/diff/1/ > > > Testing > --- > > > Thanks, > > Na Li > >
Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215475
---
Ship it!
Ship It!
- Sridhar K
On May 22, 2019, 5:36 p.m., Na Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
>
> (Updated May 22, 2019, 5:36 p.m.)
>
>
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath
> Subramanian.
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Integrate with Impala hook using Impala private jar with post exec hook
>
> changes:
>
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex
>
> {
> "id":4,
> "vertexType":"TABLE",
> "vertexId":"db_1.table_1",
> "createTime":155475
> }
>
> To embed the table create time inside column vertex
>
> {
> "id": 3,
> "vertexType": "COLUMN",
> "vertexId": "db_1.table_1.tinyint_col",
> "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
> }
> }
>
> 2) add Impala hook API code to allow Atlas integration code to compile
> without Impala updates
>
>
> Diffs
> -
>
> addons/impala-bridge-shim/pom.xml PRE-CREATION
>
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
> PRE-CREATION
> addons/impala-bridge/pom.xml 8e7e952
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
> fdb6748
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
> afe296c
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> e4a38a3
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
> 7c1103a
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
> 82672c9
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
> PRE-CREATION
>
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
> cc62955
>
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
> 6e4d321
> addons/impala-bridge/src/test/resources/impala1.json 8f747f6
> addons/impala-bridge/src/test/resources/impala2.json 239797b
> addons/impala-bridge/src/test/resources/impala3.json 6a7d171
> addons/impala-bridge/src/test/resources/impala5.json 854969b
> addons/impala-bridge/src/test/resources/impala6.json f136180
> addons/impala-bridge/src/test/resources/impala7.json f9ee670
> addons/impala-bridge/src/test/resources/impalaAlterViewAsSelect.json
> PRE-CREATION
> addons/impala-bridge/src/test/resources/impalaCreateTableAsSelect.json
> PRE-CREATION
> addons/impala-bridge/src/test/resources/impalaCreateView.json PRE-CREATION
> addons/impala-bridge/src/test/resources/impalaCreateViewNoCreateTime.json
> PRE-CREATION
> addons/impala-bridge/src/test/resources/impalaInsertIntoAsSelect.json
> PRE-CREATION
> addons/impala-hook-api/pom.xml PRE-CREATION
>
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
> PRE-CREATION
>
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java
> PRE-CREATION
> pom.xml 0a8f9b7
>
>
> Diff: https://reviews.apache.org/r/70619/diff/11/
>
>
> Testing
> ---
>
> add integration tests
> ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the
> situation that there is no createTime in lineage record
> change the file content for other integration tests in ImpalaLineageToolIT to
> handle new lineage record format
>
>
> Thanks,
>
> Na Li
>
>
Re: Review Request 70600: Adding the model files for Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70600/#review215286 --- addons/models/1000-Hadoop/1090-spark_model.json Lines 266 (patched) <https://reviews.apache.org/r/70600/#comment301950> Can we get a better descriprtion for it. In general, same comment for all similar items. addons/models/1000-Hadoop/1090-spark_model.json Lines 312 (patched) <https://reviews.apache.org/r/70600/#comment301951> date types prefered. addons/models/1000-Hadoop/1090-spark_model.json Lines 359 (patched) <https://reviews.apache.org/r/70600/#comment301952> I am assuming that there will be relationships expressing lineage from hive to spark tables and vice versa? - Sridhar K On May 15, 2019, 7:39 p.m., Aadarsh Jajodia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70600/ > --- > > (Updated May 15, 2019, 7:39 p.m.) > > > Review request for atlas, Ankur Gupta, Arun Mahadevan, Ashutosh Mestry, > Jungtaek Lim, Sridhar K, Le Ma, Madhan Neethiraj, and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > This is an initial mode file for the SPARK models. Please refer to the file > added here. This file is a GET call on typeDefs on a cluster which had SAC > enabled and SAC has created the models. The SAC version used was > https://github.com/hortonworks/spark-atlas-connector/tree/SAC-CDP-1.0-dev > > > Diffs > - > > addons/models/1000-Hadoop/1090-spark_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/70600/diff/1/ > > > Testing > --- > > > File Attachments > > > SAC Models > > https://reviews.apache.org/media/uploaded/files/2019/05/07/e31827b2-04c5-4fab-b5b7-4271398831e4__spark_sac_api_model_definition.txt > > > Thanks, > > Aadarsh Jajodia > >
Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas
> On May 9, 2019, 12:32 a.m., Sarath Subramanian wrote: > > Ship It! Let us hold on ship it. I need some clarity on this? - Sridhar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70512/#review215145 --- On May 8, 2019, 9:36 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70512/ > --- > > (Updated May 8, 2019, 9:36 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath > Subramanian, and Xinran Tinney. > > > Repository: atlas > > > Description > --- > > Impala generates lineage records for its commands. This new feature will read > Impala lineage file, convert the lineage record to Atlas entities and send > them to Atlas. In this way, Atlas can get lineage of Impala operation. > > The metadata referred in the lineage are captured in Hive Metastore hook and > sent to Atlas. This work is done in ATLAS-3148 > > This jira only supports the Impala command "create view". Following jira will > add support for more Impala commands. > > > Diffs > - > > addons/impala-bridge/pom.xml PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java > PRE-CREATION > addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-application.properties > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION > addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION > addons/impala-bridge/src/test/resources/users-credentials.properties > PRE-CREATION > addons/models/1000-Hadoop/1090-impala_model.json PRE-CREATION > pom.xml ae4dfdc > > > Diff: https://reviews.apache.org/r/70512/diff/16/ > > > Testing > --- > > Run the tool in real cluster that has Atlas server with Impala lineage file > as input for creating view. The Atlas UI displays hive_lineage lineage and > hive_column_lineage. > Add new integration tests and they pass > > > Thanks, > > Na Li > >
Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70512/#review215151 --- addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 82 (patched) <https://reviews.apache.org/r/70512/#comment301662> I might be missing some thing here. Why are we deleting the wal file? Did we have some change in the way the lineage files are generated? Normally, we used to have the lineage file continously appeneded with new lineage records. Isn't that the case any more? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 82 (patched) <https://reviews.apache.org/r/70512/#comment301663> I might be missing some thing. Why are we deleting the wal file? Did we have some change in the way the lineage files are generated? Normally, we used to have the lineage file continously appeneded with new lineage records. Isn't that the case any more? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 124 (patched) <https://reviews.apache.org/r/70512/#comment301653> This might be what is the scenario. I suggest that wqe refer the script name in the script and class name in the class. Can we please fix this? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 134 (patched) <https://reviews.apache.org/r/70512/#comment301654> Can we call it, getActiveLineageFiles() instead? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 138 (patched) <https://reviews.apache.org/r/70512/#comment301655> Do we want prefix to be user configurable? This makes the user to choose the name after using a prefix for some time and changing it to some thing different. Now, you have bunch of old files lying around. :-). addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 154 (patched) <https://reviews.apache.org/r/70512/#comment301657> Can you create a CONSTANT and reuse it? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 158 (patched) <https://reviews.apache.org/r/70512/#comment301656> Do we need this variable? Why not return the value directly? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 181 (patched) <https://reviews.apache.org/r/70512/#comment301658> importHImapala ==> importImpala name ==> fileName. addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 189 (patched) <https://reviews.apache.org/r/70512/#comment301659> Can we move these three lines into one method? On a side note, why do we even maintain a wal file, when you delete the wal file immediately after returning from this method? - Sridhar K On May 8, 2019, 9:36 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70512/ > --- > > (Updated May 8, 2019, 9:36 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath > Subramanian, and Xinran Tinney. > > > Repository: atlas > > > Description > --- > > Impala generates lineage records for its commands. This new feature will read > Impala lineage file, convert the lineage record to Atlas entities and send > them to Atlas. In this way, Atlas can get lineage of Impala operation. > > The metadata referred in the lineage are captured in Hive Metastore hook and > sent to Atlas. This work is done in ATLAS-3148 > > This jira only supports the Impala command "create view". Following jira will > add support for more Impala commands. > > > Diffs > - > > addons/impala-bridge/pom.xml PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > PRE-CREATION > > addons/impala-bridge/src/main/java/
Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70512/#review215152 --- addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java Lines 19 (patched) <https://reviews.apache.org/r/70512/#comment301665> Is the folder structure right? addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java Lines 21 (patched) <https://reviews.apache.org/r/70512/#comment301664> We have CREATEVIEW. But, not VIEW type in ImpalaVertexType. Is that right? - Sridhar K On May 8, 2019, 9:36 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70512/ > --- > > (Updated May 8, 2019, 9:36 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath > Subramanian, and Xinran Tinney. > > > Repository: atlas > > > Description > --- > > Impala generates lineage records for its commands. This new feature will read > Impala lineage file, convert the lineage record to Atlas entities and send > them to Atlas. In this way, Atlas can get lineage of Impala operation. > > The metadata referred in the lineage are captured in Hive Metastore hook and > sent to Atlas. This work is done in ATLAS-3148 > > This jira only supports the Impala command "create view". Following jira will > add support for more Impala commands. > > > Diffs > - > > addons/impala-bridge/pom.xml PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java > PRE-CREATION > addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-application.properties > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION > addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION > addons/impala-bridge/src/test/resources/users-credentials.properties > PRE-CREATION > addons/models/1000-Hadoop/1090-impala_model.json PRE-CREATION > pom.xml ae4dfdc > > > Diff: https://reviews.apache.org/r/70512/diff/16/ > > > Testing > --- > > Run the tool in real cluster that has Atlas server with Impala lineage file > as input for creating view. The Atlas UI displays hive_lineage lineage and > hive_column_lineage. > Add new integration tests and they pass > > > Thanks, > > Na Li > >
Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70512/#review215151 --- addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 82 (patched) <https://reviews.apache.org/r/70512/#comment301662> I might be missing some thing here. Why are we deleting the wal file? Did we have some change in the way the lineage files are generated? Normally, we used to have the lineage file continously appeneded with new lineage records. Isn't that the case any more? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 82 (patched) <https://reviews.apache.org/r/70512/#comment301663> I might be missing some thing. Why are we deleting the wal file? Did we have some change in the way the lineage files are generated? Normally, we used to have the lineage file continously appeneded with new lineage records. Isn't that the case any more? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 124 (patched) <https://reviews.apache.org/r/70512/#comment301653> This might be what is the scenario. I suggest that wqe refer the script name in the script and class name in the class. Can we please fix this? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 134 (patched) <https://reviews.apache.org/r/70512/#comment301654> Can we call it, getActiveLineageFiles() instead? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 138 (patched) <https://reviews.apache.org/r/70512/#comment301655> Do we want prefix to be user configurable? This makes the user to choose the name after using a prefix for some time and changing it to some thing different. Now, you have bunch of old files lying around. :-). addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 154 (patched) <https://reviews.apache.org/r/70512/#comment301657> Can you create a CONSTANT and reuse it? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 158 (patched) <https://reviews.apache.org/r/70512/#comment301656> Do we need this variable? Why not return the value directly? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 181 (patched) <https://reviews.apache.org/r/70512/#comment301658> importHImapala ==> importImpala name ==> fileName. addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 189 (patched) <https://reviews.apache.org/r/70512/#comment301659> Can we move these three lines into one method? On a side note, why do we even maintain a wal file, when you delete the wal file immediately after returning from this method? - Sridhar K On May 8, 2019, 9:36 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70512/ > --- > > (Updated May 8, 2019, 9:36 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath > Subramanian, and Xinran Tinney. > > > Repository: atlas > > > Description > --- > > Impala generates lineage records for its commands. This new feature will read > Impala lineage file, convert the lineage record to Atlas entities and send > them to Atlas. In this way, Atlas can get lineage of Impala operation. > > The metadata referred in the lineage are captured in Hive Metastore hook and > sent to Atlas. This work is done in ATLAS-3148 > > This jira only supports the Impala command "create view". Following jira will > add support for more Impala commands. > > > Diffs > - > > addons/impala-bridge/pom.xml PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > PRE-CREATION > > addons/impala-bridge/src/main/java/
Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70512/#review215151 --- addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 82 (patched) <https://reviews.apache.org/r/70512/#comment301662> I might be missing some thing here. Why are we deleting the wal file? Did we have some change in the way the lineage files are generated? Normally, we used to have the lineage file continously appeneded with new lineage records. Isn't that the case any more? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 82 (patched) <https://reviews.apache.org/r/70512/#comment301663> I might be missing some thing. Why are we deleting the wal file? Did we have some change in the way the lineage files are generated? Normally, we used to have the lineage file continously appeneded with new lineage records. Isn't that the case any more? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 124 (patched) <https://reviews.apache.org/r/70512/#comment301653> This might be what is the scenario. I suggest that wqe refer the script name in the script and class name in the class. Can we please fix this? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 134 (patched) <https://reviews.apache.org/r/70512/#comment301654> Can we call it, getActiveLineageFiles() instead? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 138 (patched) <https://reviews.apache.org/r/70512/#comment301655> Do we want prefix to be user configurable? This makes the user to choose the name after using a prefix for some time and changing it to some thing different. Now, you have bunch of old files lying around. :-). addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 154 (patched) <https://reviews.apache.org/r/70512/#comment301657> Can you create a CONSTANT and reuse it? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 158 (patched) <https://reviews.apache.org/r/70512/#comment301656> Do we need this variable? Why not return the value directly? addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 181 (patched) <https://reviews.apache.org/r/70512/#comment301658> importHImapala ==> importImpala name ==> fileName. addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java Lines 189 (patched) <https://reviews.apache.org/r/70512/#comment301659> Can we move these three lines into one method? On a side note, why do we even maintain a wal file, when you delete the wal file immediately after returning from this method? - Sridhar K On May 8, 2019, 9:36 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70512/ > --- > > (Updated May 8, 2019, 9:36 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath > Subramanian, and Xinran Tinney. > > > Repository: atlas > > > Description > --- > > Impala generates lineage records for its commands. This new feature will read > Impala lineage file, convert the lineage record to Atlas entities and send > them to Atlas. In this way, Atlas can get lineage of Impala operation. > > The metadata referred in the lineage are captured in Hive Metastore hook and > sent to Atlas. This work is done in ATLAS-3148 > > This jira only supports the Impala command "create view". Following jira will > add support for more Impala commands. > > > Diffs > - > > addons/impala-bridge/pom.xml PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > PRE-CREATION > > addons/impala-bridge/src/main/java/
Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70599/#review215108 --- common/src/main/java/org/apache/atlas/repository/Constants.java Lines 41 (patched) <https://reviews.apache.org/r/70599/#comment301558> I understand what you are doing this for? But, do we have to replace the existing id? Why not keep using the existing id? - Sridhar K On May 8, 2019, 3:10 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70599/ > --- > > (Updated May 8, 2019, 3:10 p.m.) > > > Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath > Subramanian. > > > Bugs: ATLAS-3193 > https://issues.apache.org/jira/browse/ATLAS-3193 > > > Repository: atlas > > > Description > --- > > **Background** > Please look at bug description. > > **Approach** > _ExportService_: Refactor to extract starting entity logic in a separate > reusable class. > (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start > entity. > _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing > entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to > check and fetch existing entities). > _GraphBackedSearchIndexer_: Additional property to index. > > **CURL** > > Import: > ``` > curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H > "Cache-Control: no-cache" -F data=@./stocks.zip > http://localhost:21000/api/atlas/admin/import > ``` > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 1cfa202a9 > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java > 62298f9f1 > > repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java > a52c3f986 > > repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java > dc8bd19df > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 14f991697 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > a1d6cef5c > > repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java > e432d4e42 > > > Diff: https://reviews.apache.org/r/70599/diff/6/ > > > Testing > --- > > **Unit tests** > New tests added. > > **Functional tests** > Existing export and import flow via CURL calls. > > **Pre-commit** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/ > > > Thanks, > > Ashutosh Mestry > >
Re: Review Request 70605: ATLAS-3197 Add QueryText for hive_table and hive_db
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70605/#review215114 --- addons/models/1000-Hadoop/1030-hive_model.json Lines 535 (patched) <https://reviews.apache.org/r/70605/#comment301578> Thanks for moving it to right file. - Sridhar K On May 7, 2019, 6:35 p.m., Le Ma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70605/ > --- > > (Updated May 7, 2019, 6:35 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3197 > https://issues.apache.org/jira/browse/ATLAS-3197 > > > Repository: atlas > > > Description > --- > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing DDLQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attribute will be deleted as > well. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java > d3d83490f > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > de8de63b6 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java > b01f61f76 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java > b5ce09332 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 24de30f5a > addons/models/1000-Hadoop/1030-hive_model.json ee6945a77 > > > Diff: https://reviews.apache.org/r/70605/diff/3/ > > > Testing > --- > > 1. table : > > CREATE TABLE IF NOT EXISTS testDDL (d int, n string); > > > ALTER TABLE testDDL RENAME TO renameTestDDL; > > ALTER TABLE renameTestDDL CHANGE n changedCol string; > > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int); > > DROP TABLE renameTestDDL; > > > 2. db > > create database testdb; > > ALTER DATABASE database SET OWNER USER changedOwner; > > DROP DATABSE testdb; > > > File Attachments > > > CREATE TABLE IF NOT EXISTS testDDL.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/5047d461-f288-4776-a843-556e45217fef__CREATE_TABLE_IF_NOT_EXISTS_testDDL.png > ALTER TABLE testDDL RENAME TO renameTestDDL; .png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/373a656d-8ab3-4710-b9a1-3eac20adc947__ALTER_TABLE_testDDL_RENAME_TO_renameTestDDL_.png > ALTER TABLE renameTestDDL CHANGE n changedCol string; - ddl.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/c9f57d67-c16b-4142-a6b4-9d372e7fd765__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string_-_ddl.png > ALTER TABLE renameTestDDL CHANGE n changedCol string;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/42612eb6-83c2-4825-8601-5a1629c9de6f__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string.png > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int);.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/f5d3522d-17b6-4227-8072-6fc07d88d515__ALTER_TABLE_renameTestDDL_ADD_COLUMNSaddedCol_int.png > ALTER TABLE renameTestDDL ADD COLUMNS.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/73b9046a-2667-4049-9e11-c5c2b375e9ca__ALTER_TABLE_renameTestDDL_ADD_COLUMNS.png > DROP TABLE renameTestDDL;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/ebfa01f2-e886-4f94-aa75-ff45bb519ca0__DROP_TABLE_renameTestDDL.png > ddl - relationship.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/7b30282b-6419-4791-b107-f938918e1cfd__ddl_-_relationship.png > ALTER DATABASE testdb SET OWNER USER changedOwner;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/388e7ffa-6691-4c2b-bcd9-d8755dc13e3e__ALTER_DATABASE_testdb_SET_OWNER_USER_changedOwner.png > ALTER DATABASE testdb SET OWNER USER changedOwner; - db entity.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/fd22b6a5-6cb9-40eb-aa10-538
Re: Review Request 70605: ATLAS-3197 Add QueryText for hive_table and hive_db
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70605/#review215113 --- addons/models/1000-Hadoop/1030-hive_model.json Lines 586 (patched) <https://reviews.apache.org/r/70605/#comment301577> Too generic name. Can we get a btter name here. I suggest execTime. Or is it one of those name consistency issue? - Sridhar K On May 7, 2019, 6:35 p.m., Le Ma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70605/ > --- > > (Updated May 7, 2019, 6:35 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3197 > https://issues.apache.org/jira/browse/ATLAS-3197 > > > Repository: atlas > > > Description > --- > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing DDLQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attribute will be deleted as > well. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java > d3d83490f > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > de8de63b6 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java > b01f61f76 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java > b5ce09332 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 24de30f5a > addons/models/1000-Hadoop/1030-hive_model.json ee6945a77 > > > Diff: https://reviews.apache.org/r/70605/diff/3/ > > > Testing > --- > > 1. table : > > CREATE TABLE IF NOT EXISTS testDDL (d int, n string); > > > ALTER TABLE testDDL RENAME TO renameTestDDL; > > ALTER TABLE renameTestDDL CHANGE n changedCol string; > > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int); > > DROP TABLE renameTestDDL; > > > 2. db > > create database testdb; > > ALTER DATABASE database SET OWNER USER changedOwner; > > DROP DATABSE testdb; > > > File Attachments > > > CREATE TABLE IF NOT EXISTS testDDL.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/5047d461-f288-4776-a843-556e45217fef__CREATE_TABLE_IF_NOT_EXISTS_testDDL.png > ALTER TABLE testDDL RENAME TO renameTestDDL; .png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/373a656d-8ab3-4710-b9a1-3eac20adc947__ALTER_TABLE_testDDL_RENAME_TO_renameTestDDL_.png > ALTER TABLE renameTestDDL CHANGE n changedCol string; - ddl.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/c9f57d67-c16b-4142-a6b4-9d372e7fd765__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string_-_ddl.png > ALTER TABLE renameTestDDL CHANGE n changedCol string;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/42612eb6-83c2-4825-8601-5a1629c9de6f__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string.png > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int);.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/f5d3522d-17b6-4227-8072-6fc07d88d515__ALTER_TABLE_renameTestDDL_ADD_COLUMNSaddedCol_int.png > ALTER TABLE renameTestDDL ADD COLUMNS.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/73b9046a-2667-4049-9e11-c5c2b375e9ca__ALTER_TABLE_renameTestDDL_ADD_COLUMNS.png > DROP TABLE renameTestDDL;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/ebfa01f2-e886-4f94-aa75-ff45bb519ca0__DROP_TABLE_renameTestDDL.png > ddl - relationship.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/7b30282b-6419-4791-b107-f938918e1cfd__ddl_-_relationship.png > ALTER DATABASE testdb SET OWNER USER changedOwner;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/388e7ffa-6691-4c2b-bcd9-d8755dc13e3e__ALTER_DATABASE_testdb_SET_OWNER_USER_changedOwner.png > ALTER DATABASE testdb SET OWNER USER changedOwner; - db entity.png > > https://reviews.apache.org/media/upl
Re: Review Request 70605: ATLAS-3197 Add QueryText for hive_table and hive_db
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70605/#review215107 --- addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java Lines 669 (patched) <https://reviews.apache.org/r/70605/#comment301552> Can we call it createHiveDDLEntity() instead? We are creating an object here? :-) addons/models/-Area0/0010-base_model.json Lines 388 (patched) <https://reviews.apache.org/r/70605/#comment301550> Can we give more meaningful name? addons/models/-Area0/0010-base_model.json Lines 471 (patched) <https://reviews.apache.org/r/70605/#comment301553> Your model is right from semantics of our ddl entitites. You are modeling this as composition. I am a bit concerned with this idea. Do we know how atlas/janus would treat the semantic violations? I am looking at already created databases/tables. We don't have ddls linked for them? So, will Janus be alright with data inconsistency like this. Something to watch out for. The scenario I am talking about is "upgrade scenario". - Sridhar K On May 7, 2019, 6:35 p.m., Le Ma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70605/ > --- > > (Updated May 7, 2019, 6:35 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3197 > https://issues.apache.org/jira/browse/ATLAS-3197 > > > Repository: atlas > > > Description > --- > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing DDLQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attribute will be deleted as > well. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java > d3d83490f > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > de8de63b6 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java > b01f61f76 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java > b5ce09332 > addons/models/-Area0/0010-base_model.json d14b2270e > > > Diff: https://reviews.apache.org/r/70605/diff/1/ > > > Testing > --- > > 1. table : > > CREATE TABLE IF NOT EXISTS testDDL (d int, n string); > > > ALTER TABLE testDDL RENAME TO renameTestDDL; > > ALTER TABLE renameTestDDL CHANGE n changedCol string; > > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int); > > DROP TABLE renameTestDDL; > > > 2. db > > create database testdb; > > ALTER DATABASE database SET OWNER USER changedOwner; > > DROP DATABSE testdb; > > > File Attachments > > > CREATE TABLE IF NOT EXISTS testDDL.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/5047d461-f288-4776-a843-556e45217fef__CREATE_TABLE_IF_NOT_EXISTS_testDDL.png > ALTER TABLE testDDL RENAME TO renameTestDDL; .png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/373a656d-8ab3-4710-b9a1-3eac20adc947__ALTER_TABLE_testDDL_RENAME_TO_renameTestDDL_.png > ALTER TABLE renameTestDDL CHANGE n changedCol string; - ddl.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/c9f57d67-c16b-4142-a6b4-9d372e7fd765__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string_-_ddl.png > ALTER TABLE renameTestDDL CHANGE n changedCol string;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/42612eb6-83c2-4825-8601-5a1629c9de6f__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string.png > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int);.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/f5d3522d-17b6-4227-8072-6fc07d88d515__ALTER_TABLE_renameTestDDL_ADD_COLUMNSaddedCol_int.png > ALTER TABLE renameTestDDL ADD COLUMNS.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/73b9046a-2667-4049-9e11-c5c2b375e9ca__ALTER_TABLE_renameTestDDL_ADD_COLUMNS.png > DROP TAB
Re: Review Request 70600: Adding the model files for Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70600/#review215100 --- addons/models/1000-Hadoop/1090-spark_model.json Lines 7 (patched) <https://reviews.apache.org/r/70600/#comment301525> I am assuming that we are going to reconsider these models. Especially, there is a possibility of these objects are represented as hive or habase objects in Atlas already. - Sridhar K On May 7, 2019, 8:01 p.m., Aadarsh Jajodia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70600/ > --- > > (Updated May 7, 2019, 8:01 p.m.) > > > Review request for atlas, Ashutosh Mestry, Jungtaek Lim, Sridhar K, Le Ma, > Madhan Neethiraj, and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > This is an initial mode file for the SPARK models. Please refer to the file > added here. This file is a GET call on typeDefs on a cluster which had SAC > enabled and SAC has created the models. The SAC version used was > https://github.com/hortonworks/spark-atlas-connector/tree/SAC-CDP-1.0-dev > > > Diffs > - > > addons/models/1000-Hadoop/1090-spark_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/70600/diff/1/ > > > Testing > --- > > > File Attachments > > > SAC Models > > https://reviews.apache.org/media/uploaded/files/2019/05/07/e31827b2-04c5-4fab-b5b7-4271398831e4__spark_sac_api_model_definition.txt > > > Thanks, > > Aadarsh Jajodia > >
Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70512/#review215076 --- My review is still pending. thanks, -Sridhar addons/impala-bridge/pom.xml Lines 65 (patched) <https://reviews.apache.org/r/70512/#comment301477> Can we please move this version to a variable? addons/impala-bridge/pom.xml Lines 71 (patched) <https://reviews.apache.org/r/70512/#comment301476> Can we please move this version to a variable? Use the same variable as above. addons/impala-bridge/pom.xml Lines 449 (patched) <https://reviews.apache.org/r/70512/#comment301478> Please move to variable. - Sridhar K On May 4, 2019, 9:23 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70512/ > --- > > (Updated May 4, 2019, 9:23 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath > Subramanian, and Xinran Tinney. > > > Repository: atlas > > > Description > --- > > Impala generates lineage records for its commands. This new feature will read > Impala lineage file, convert the lineage record to Atlas entities and send > them to Atlas. In this way, Atlas can get lineage of Impala operation. > > The metadata referred in the lineage are captured in Hive Metastore hook and > sent to Atlas. This work is done in ATLAS-3148 > > This jira only supports the Impala command "create view". Following jira will > add support for more Impala commands. > > > Diffs > - > > addons/impala-bridge/pom.xml PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java > PRE-CREATION > addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-application.properties > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION > addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION > addons/impala-bridge/src/test/resources/users-credentials.properties > PRE-CREATION > addons/models/1000-Hadoop/1090-impala_mod
Re: Review Request 70588: ATLAS-3188: Regression: Simple insert queries are being captured in ATLAS_HOOK topic
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70588/#review215045 --- addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java Lines 122 (patched) <https://reviews.apache.org/r/70588/#comment301424> Why do we need this check...just assign the value. addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java Lines 128 (patched) <https://reviews.apache.org/r/70588/#comment301425> same here. - Sridhar K On May 3, 2019, 4:36 a.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70588/ > --- > > (Updated May 3, 2019, 4:36 a.m.) > > > Review request for atlas, Ashutosh Mestry and Madhan Neethiraj. > > > Bugs: ATLAS-3188 > https://issues.apache.org/jira/browse/ATLAS-3188 > > > Repository: atlas > > > Description > --- > > insert into values () is triggering messages in ATLAS_HOOK > topic. Due to this, Atlas is reporting the operations in Audits. > > Ideally, insert queries should not generate any messages in ATLAS_HOOK since > they do not play any role in lineage > > Inserting 100 rows in a table is generating 100 messages in the kafka topic. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java > 54cd48f04 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > f378d8457 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java > cdb4cd218 > > > Diff: https://reviews.apache.org/r/70588/diff/2/ > > > Testing > --- > > validated that "insert into values" query doesn't create hive_process entities > > > Thanks, > > Sarath Subramanian > >
