[jira] [Commented] (ATLAS-1955) Validation for Attributes

2017-08-17 Thread Israel Varea (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130253#comment-16130253
 ] 

Israel Varea commented on ATLAS-1955:
-

I can't find a strong argument to decide between the two options. A good 
question to choose between the two options is:  must validations be reusable 
for different attributes?
If the answer is yes, then go with Option 2. If the answer is "A reusable 
validation is not so important" then go with Option 1 since it is more simple.
I think both options will cover most of the common use cases anyway.



> Validation for Attributes
> -
>
> Key: ATLAS-1955
> URL: https://issues.apache.org/jira/browse/ATLAS-1955
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Israel Varea
>Assignee: Richard Ding
> Fix For: 0.9-incubating
>
>
> It would be very nice that Atlas model could contain a way to represent 
> attribute validation. 
> A simple example is that we would like to model a Person, with attributes 
> Name, Email and Country. Now we would like to specify that Email has to 
> follow a specific regular expression, so it would be nice if we could set 
> Email -> hasValidation -> EmailRegex, with EmailRegex having:
> Name: Email Regular Expresion
> Expression: /[0-9a-z]+@[0-9a-z]+.[0-9a-z]+/
> For more complex types of validation, e.g. checking card number validity, it 
> could be added some external validator function/service.
> Name: Credit Card Number Validator
> Validator: org.apache.atlas.validators.creditcard or 
> https://host:port/creditCardValidator
> For validations from a reference table, for example a country name, it could 
> be:
> Name: Country Name Ref Validator
> Reference Column: 
> where  would be an instance of type Hive_Column or 
> HBase_Column.
> Since this is a kind of Standarization, it could be placed in [Area 
> 5|https://cwiki.apache.org/confluence/display/ATLAS/Area+5+-+Standards].
> A similar approach is followed in software 
> [Kylo|https://github.com/Teradata/kylo/tree/master/integrations/spark/spark-validate-cleanse]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (ATLAS-1955) Validation for Attributes

2017-07-24 Thread Israel Varea (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098207#comment-16098207
 ] 

Israel Varea commented on ATLAS-1955:
-

I think these two validations will cover most of the basic validation cases.
In the second validation, from a reference table, it would be nice to be able 
to provide a reference to a column of a table, since columns are already 
modeled. I think importing a reference table into Atlas enums will duplicate 
data, and you will have to keep a synchronization between the two of them, so I 
think it will be much simple if we just point to a column. However, both of the 
two alternatives can solve succesfully the same modelling use case :)


> Validation for Attributes
> -
>
> Key: ATLAS-1955
> URL: https://issues.apache.org/jira/browse/ATLAS-1955
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Israel Varea
> Fix For: 0.9-incubating
>
>
> It would be very nice that Atlas model could contain a way to represent 
> attribute validation. 
> A simple example is that we would like to model a Person, with attributes 
> Name, Email and Country. Now we would like to specify that Email has to 
> follow a specific regular expression, so it would be nice if we could set 
> Email -> hasValidation -> EmailRegex, with EmailRegex having:
> Name: Email Regular Expresion
> Expression: /[0-9a-z]+@[0-9a-z]+.[0-9a-z]+/
> For more complex types of validation, e.g. checking card number validity, it 
> could be added some external validator function/service.
> Name: Credit Card Number Validator
> Validator: org.apache.atlas.validators.creditcard or 
> https://host:port/creditCardValidator
> For validations from a reference table, for example a country name, it could 
> be:
> Name: Country Name Ref Validator
> Reference Column: 
> where  would be an instance of type Hive_Column or 
> HBase_Column.
> Since this is a kind of Standarization, it could be placed in [Area 
> 5|https://cwiki.apache.org/confluence/display/ATLAS/Area+5+-+Standards].
> A similar approach is followed in software 
> [Kylo|https://github.com/Teradata/kylo/tree/master/integrations/spark/spark-validate-cleanse]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (ATLAS-1839) Area 2 of the open metadata model

2017-07-20 Thread Israel Varea (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094313#comment-16094313
 ] 

Israel Varea commented on ATLAS-1839:
-

I can think in a more complex validation when it depends on a different 
attribute. A simple example is when we want to model an Event, with two 
attributes, start_time, end_time. The validation of end_time should be that 
start_time <= end_time.

However, the most common are pattern matching by regular expressions and 
validations from a reference table columns (e.g. country codes).

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (ATLAS-1955) Validation for Attributes

2017-07-17 Thread Israel Varea (JIRA)
Israel Varea created ATLAS-1955:
---

 Summary: Validation for Attributes
 Key: ATLAS-1955
 URL: https://issues.apache.org/jira/browse/ATLAS-1955
 Project: Atlas
  Issue Type: New Feature
  Components:  atlas-core
Affects Versions: 0.9-incubating
Reporter: Israel Varea
 Fix For: 0.9-incubating


It would be very nice that Atlas model could contain a way to represent 
attribute validation. 

A simple example is that we would like to model a Person, with attributes Name, 
Email and Country. Now we would like to specify that Email has to follow a 
specific regular expression, so it would be nice if we could set Email -> 
hasValidation -> EmailRegex, with EmailRegex having:

Name: Email Regular Expresion
Expression: /[0-9a-z]+@[0-9a-z]+.[0-9a-z]+/

For more complex types of validation, e.g. checking card number validity, it 
could be added some external validator function/service.

Name: Credit Card Number Validator
Validator: org.apache.atlas.validators.creditcard or 
https://host:port/creditCardValidator

For validations from a reference table, for example a country name, it could be:

Name: Country Name Ref Validator
Reference Column: 

where  would be an instance of type Hive_Column or 
HBase_Column.

Since this is a kind of Standarization, it could be placed in [Area 
5|https://cwiki.apache.org/confluence/display/ATLAS/Area+5+-+Standards].

A similar approach is followed in software 
[Kylo|https://github.com/Teradata/kylo/tree/master/integrations/spark/spark-validate-cleanse]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (ATLAS-1839) Area 2 of the open metadata model

2017-07-17 Thread Israel Varea (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089814#comment-16089814
 ] 

Israel Varea commented on ATLAS-1839:
-

Hi all,

I have been looking at how open metadata model is being defined and I think 
this iniciative is awesome.

In particular, I would like to model some domain using this Area 2 definitions, 
but I can't find a proper way to do so, may be you can have some insights of 
how to do it.

I want to model a Person, so I would include Person as a Term that inherits 
from Catalog. Now I want to add some attributes (concepts) related to a Person, 
such as Person_Name, Person_Surname, Date_of_Birth, Email, so I add them as 
Terms too.
How should I relate these Terms as "attribute_of" Person? I see there is a 0250 
Related Terms section, however there is no relation to specify "attribute_of" 
or similar.

In this model, I would like to specify some features for Terms, such as 
validation/normalization, max and min values, etc. For example, I would like to 
specify that only a specific regex is valid for Emails, or Card_Numbers. 
How could I add these features for the Terms?

Thanks




> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0210Glossary.json, 0220CategoryHierarchy.json, 
> 0230Terms.json, 0240Dictionary.json, 0250RelatedTerms.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)