[jira] [Commented] (ATLAS-1839) Area 2 of the open metadata model
[ https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094613#comment-16094613 ] Nigel Jones commented on ATLAS-1839: [~davidrad] the toxic combination support in ranger policies is primarily geared to controlling what a user may access, whilst the validation [~ivarea] is suggesting is primarily about creates and updates, ie defining the data model itself. That's not to say ranger couldn't do this (since it can address any operation such as a create) but I don't think that's ranger's intent. But I agree it's a fine line and could well vary significantly in different environments As such I think it makes sense to define validation in atlas and be able to link to code artifacts, services that implement those validations probably through a combination of discovery & stewardship , plus making it easier when writing pipelines for say ETL or streaming, to be able to easily pull in atlas metadata and capture a link between a validation implemented by an pipeline author (or being used from a library) and it's definition in atlas. Thus atlas ends up with both the "intent" (the business spec if you like) as well as links to the implementation yet does not constrain those implementations since they can be so varied. Following on from this, absolutely some of those validations could be implemented as complex rules, but I think it would be tricky and constraining to capture all that in atlas, hence why I'd go for the link approach & some relatively loose coupling So with that done, sure we could have a more complex rules engine embedded in, or used by ranger plugins... but this could be one of a number of different approaches I'd be inclined to start off with us figuring out how to model, and some use cases where we can explore the authoring (ie in atlas), assisted authoring (when writing a job), metadata capture (from those other systems, also relates to lineage) & probably best to do that in ATLAS-1995? This also touches on RANGER-1869 (metadata capture) Certainly this is an interesting area ! > Area 2 of the open metadata model > - > > Key: ATLAS-1839 > URL: https://issues.apache.org/jira/browse/ATLAS-1839 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 0.9-incubating >Reporter: Mandy Chessell >Assignee: David Radley > Labels: OpenMetadata, VirtualDataConnector > Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, > 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, > 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, > 0280SpineObjects.json > > > This task delivers the JSON files for the new models that describe types for > Area 2 in the open metadata model. This area covers the glossary. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1839) Area 2 of the open metadata model
[ https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094313#comment-16094313 ] Israel Varea commented on ATLAS-1839: - I can think in a more complex validation when it depends on a different attribute. A simple example is when we want to model an Event, with two attributes, start_time, end_time. The validation of end_time should be that start_time <= end_time. However, the most common are pattern matching by regular expressions and validations from a reference table columns (e.g. country codes). > Area 2 of the open metadata model > - > > Key: ATLAS-1839 > URL: https://issues.apache.org/jira/browse/ATLAS-1839 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 0.9-incubating >Reporter: Mandy Chessell >Assignee: David Radley > Labels: OpenMetadata, VirtualDataConnector > Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, > 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, > 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, > 0280SpineObjects.json > > > This task delivers the JSON files for the new models that describe types for > Area 2 in the open metadata model. This area covers the glossary. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1839) Area 2 of the open metadata model
[ https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089814#comment-16089814 ] Israel Varea commented on ATLAS-1839: - Hi all, I have been looking at how open metadata model is being defined and I think this iniciative is awesome. In particular, I would like to model some domain using this Area 2 definitions, but I can't find a proper way to do so, may be you can have some insights of how to do it. I want to model a Person, so I would include Person as a Term that inherits from Catalog. Now I want to add some attributes (concepts) related to a Person, such as Person_Name, Person_Surname, Date_of_Birth, Email, so I add them as Terms too. How should I relate these Terms as "attribute_of" Person? I see there is a 0250 Related Terms section, however there is no relation to specify "attribute_of" or similar. In this model, I would like to specify some features for Terms, such as validation/normalization, max and min values, etc. For example, I would like to specify that only a specific regex is valid for Emails, or Card_Numbers. How could I add these features for the Terms? Thanks > Area 2 of the open metadata model > - > > Key: ATLAS-1839 > URL: https://issues.apache.org/jira/browse/ATLAS-1839 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 0.9-incubating >Reporter: Mandy Chessell >Assignee: Mandy Chessell > Labels: OpenMetadata, VirtualDataConnector > Attachments: 0210Glossary.json, 0220CategoryHierarchy.json, > 0230Terms.json, 0240Dictionary.json, 0250RelatedTerms.json > > > This task delivers the JSON files for the new models that describe types for > Area 2 in the open metadata model. This area covers the glossary. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1839) Area 2 of the open metadata model
[ https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068811#comment-16068811 ] Mandy Chessell commented on ATLAS-1839: --- I will change "Taxomony" to "IsTaxonomy" Source will have to be a string because structs are not allowed in the relationship attributes. Using capitalization has no influence from SQL. They are used on the relationship names to make the type of relationship stand out. When mixed case they were difficult to read. > Area 2 of the open metadata model > - > > Key: ATLAS-1839 > URL: https://issues.apache.org/jira/browse/ATLAS-1839 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 0.9-incubating >Reporter: Mandy Chessell >Assignee: Mandy Chessell > Labels: OpenMetadata, VirtualDataConnector > Attachments: 0210Glossary.json, 0220CategoryHierarchy.json, > 0230Terms.json, 0240Dictionary.json, 0250RelatedTerms.json > > > This task delivers the JSON files for the new models that describe types for > Area 2 in the open metadata model. This area covers the glossary. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
