Hi all, I have been having an email conversation with William (Songqing Ding). William is looking for some Atlas work to do. I suggested he looks at introducing new AttributeDefs that allow strings to be validated using regex patterns; Richard has responded with other helpful considerations. This requirement is something that Madhan and others were supportive of in a meeting a couple of weeks ago.
The below is the email exchange we have had to date. This email is to invite the community to contribute to this design discussion, many thanks, David. ----- Forwarded by David Radley/UK/IBM on 08/08/2017 10:12 ----- From: David Radley/UK/IBM To: Songqing Ding/Silicon Valley/IBM@IBMUS Cc: Graham Wallis/UK/IBM@IBMGB, Kelvin Lawrence/Austin/IBM@IBMUS, Mandy Chessell/UK/IBM@IBMGB, Nigel L Jones/UK/IBM@IBMGB Date: 08/08/2017 10:12 Subject: Re: Fw: Atlas Jiras and features Hi Richard, Some comments in <DAVID> tags. all the best David. From: Songqing Ding/Silicon Valley/IBM To: David Radley/UK/IBM@IBMGB Cc: Graham Wallis/UK/IBM@IBMGB, Kelvin Lawrence/Austin/IBM@IBMUS, Mandy Chessell/UK/IBM@IBMGB, Nigel L Jones/UK/IBM@IBMGB Date: 08/08/2017 00:46 Subject: Re: Fw: Atlas Jiras and features Hi David, Thanks for the info, it is very helpful. I took a look at these Jiras: ATLAS-1955: Validation for Attributes Each Atlas Type (e.g. String, Date, Struct, ...) currently has a isValidValue(Object obj) method and attribute values are validated using attribute types. So if Atlas defines a static EMail type, he or she can implements isValidValue to validate the email addresses. <DAVID> My thinking was that we should make the new attribtutedef validation, data driven and declarative and not require programming. If people want very complex validation that cannot be easily represented as a pattern - then they could look at the programming approach. I think we open up the ability to add custom validation - if all they need to do is understand regex. Defining the validation in metadata also means that it is managed with the other metadata like EntityDefs, whereas managing custom code fragments that may not be in the master would become a headache. </DAVID> If, say, different organizations have different email formats, then one wants to define dynamic email attributes. The existing facility on can use is AtlasConstraintDef. Currently each AtlasAttributeDef contains a list of AtlasConstraintDefs. However only two types of constraints (ownedRef & inverseRef) are defined and constraints are not used to validate the attribute values. Therefore, one way to implement a pattern matching validation for attributes is to add a PATTERN_MATCH constraint type and check the constraint during the value validation process. <DAVID> I suggest we have a basic email pattern, which looks for characters followed by an @ followed by string segments separated by dots. If an organization wants to have a more specialized or different mail format - for example a Lotus notes format - they just define a new type with a different pattern. With the proposed approach, we supply a attributedef that 90% could take and the other 10% have the flexibility to create their own versions if they want to. </DAVID> The drawback of the 2nd approach is that the constraints are local to the entity types. One needs to define the same constraint in multiple entity types if needed. ATLAS-1953: It is good idea to have icon attributes so that users can define icons associated with their entity types. An new icon type can encapsulate this requirement. Thanks, -Richard ----- Original message ----- From: David Radley/UK/IBM To: Kelvin Lawrence/Austin/IBM@IBMUS Cc: Graham Wallis/UK/IBM@IBMGB, Mandy Chessell/UK/IBM@IBMGB, Songqing Ding/Silicon Valley/IBM@IBMUS, Nigel L Jones/UK/IBM@IBMGB Subject: Fw: Atlas Jiras and features Date: Mon, Aug 7, 2017 4:05 AM Hi Kelvin, I am back now. I noticed Richard has been active on the Jiras. I have an idea that Richard could look at; in the last call I attended, I mentioned some features, that would be great to get in, which the team supported. I was thinking Richard could do something along the lines of (draft unreviewed design follows .... :-) ) : 1) Adding an optional icon to EntityDefs 2) To do this is would be good to have a new ValidatableStringDefs that could police the value of the attribute, the idea is that ValidatableStringDefs could be CRUD in a similar way to EntityDefs. These ValidatableStringDefs would have mandatory Regex expressions and possibly a customer validation message. 3) New ValidatableAttributeDefs would have have specific Regex expressions in. Atlas can ship email, url, image AttributeDefs (and others?) . 4) The optional icon in EntityDef could then have a type IMAGE. 5) Due to the inability to validate and reject content during imports - we are looking to accept string content - and return warning not error return codes indicating that a value was not the expected format. I recall we talked of a Kafka message being issued to indicate this has occurred. 6) This will be useful for the new extended types; as we/he could add icons for terms and categories and the like that UI apps could then use to should a much nicer UI. I think this would be reasonably contained function to bring Richard into the project; this work could be split up into a sequence of small reviewable tasks. Existing Jiras in this space are ATLAS-1953. and ATLAS-1955. If this appeals to Richard - I could send this note copying the dev list - so the community can feedback and Madhan is aware - in case HW have / are thinking of picking this up. all the best, David. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU