[ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094613#comment-16094613
 ] 

Nigel Jones edited comment on ATLAS-1839 at 7/20/17 12:37 PM:
--------------------------------------------------------------

[~davidrad] the toxic combination support in ranger policies is primarily 
geared to controlling what a user may access, whilst the validation [~ivarea] 
is suggesting is primarily about creates and updates, ie defining the data 
model itself. That's not to say ranger couldn't do this (since it can address 
any operation such as a create) but I don't think that's ranger's intent. But I 
agree it's a fine line and could well vary significantly in different 
environments

As such I think it makes sense to define validation in atlas and be able to 
link to code artifacts, services that implement those validations probably 
through a combination of discovery & stewardship , plus making it easier when 
writing pipelines for say ETL or streaming, to be able to easily pull in atlas 
metadata and capture a link between a validation implemented by an pipeline 
author (or being used from a library) and it's definition in atlas. Thus atlas 
ends up with both the "intent" (the business spec if you like) as well as links 
to the implementation yet does not constrain those implementations since they 
can be so varied. It's also incremental, easy to adopt, and means one can get 
value without everything being in place?

Following on from this, absolutely some of those validations could be 
implemented as complex rules using a full featured rules engine, but I think it 
would be tricky and constraining to capture all that in atlas, hence why I'd go 
for the link approach & some relatively loose coupling

So with that done, sure we could have a more complex rules engine embedded in, 
or used by ranger plugins... but this could be one of a number of different 
approaches

I'd be inclined to start off with us figuring out how to model, and some use 
cases where we can explore the authoring (ie in atlas), assisted authoring 
(when writing a job), metadata capture (from those other systems, also relates 
to lineage) & probably best to do that in ATLAS-1995? This also touches on 
RANGER-1869 (metadata capture)

Certainly this is an interesting area !


was (Author: jonesn):
[~davidrad] the toxic combination support in ranger policies is primarily 
geared to controlling what a user may access, whilst the validation [~ivarea] 
is suggesting is primarily about creates and updates, ie defining the data 
model itself. That's not to say ranger couldn't do this (since it can address 
any operation such as a create) but I don't think that's ranger's intent. But I 
agree it's a fine line and could well vary significantly in different 
environments

As such I think it makes sense to define validation in atlas and be able to 
link to code artifacts, services that implement those validations probably 
through a combination of discovery & stewardship , plus making it easier when 
writing pipelines for say ETL or streaming, to be able to easily pull in atlas 
metadata and capture a link between a validation implemented by an pipeline 
author (or being used from a library) and it's definition in atlas. Thus atlas 
ends up with both the "intent" (the business spec if you like) as well as links 
to the implementation yet does not constrain those implementations since they 
can be so varied. 

Following on from this, absolutely some of those validations could be 
implemented as complex rules using a full featured rules engine, but I think it 
would be tricky and constraining to capture all that in atlas, hence why I'd go 
for the link approach & some relatively loose coupling

So with that done, sure we could have a more complex rules engine embedded in, 
or used by ranger plugins... but this could be one of a number of different 
approaches

I'd be inclined to start off with us figuring out how to model, and some use 
cases where we can explore the authoring (ie in atlas), assisted authoring 
(when writing a job), metadata capture (from those other systems, also relates 
to lineage) & probably best to do that in ATLAS-1995? This also touches on 
RANGER-1869 (metadata capture)

Certainly this is an interesting area !

> Area 2 of the open metadata model
> ---------------------------------
>
>                 Key: ATLAS-1839
>                 URL: https://issues.apache.org/jira/browse/ATLAS-1839
>             Project: Atlas
>          Issue Type: Task
>          Components:  atlas-core
>    Affects Versions: 0.9-incubating
>            Reporter: Mandy Chessell
>            Assignee: David Radley
>              Labels: OpenMetadata, VirtualDataConnector
>         Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to