[jira] [Updated] (ATLAS-3601) No LABEL_DELETE event is audited when last label is deleted

2020-02-02 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3601:
--
Affects Version/s: 2.0.0

>  No LABEL_DELETE event is audited when last label is deleted
> 
>
> Key: ATLAS-3601
> URL: https://issues.apache.org/jira/browse/ATLAS-3601
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
> Attachments: ATLAS-3601.patch
>
>
> When 1 label among multiple labels is removed , LABEL_DELETE is audited. But 
> when all labels are deleted or the last label is deleted , no LABEL_DELETE is 
> audited.
> Also , we see "One or more existing labels were removed successfully" popup 
> when there is no LABEL_DELETE audit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3601) No LABEL_DELETE event is audited when last label is deleted

2020-02-02 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3601:
--
Component/s:  atlas-core

>  No LABEL_DELETE event is audited when last label is deleted
> 
>
> Key: ATLAS-3601
> URL: https://issues.apache.org/jira/browse/ATLAS-3601
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
> Attachments: ATLAS-3601.patch
>
>
> When 1 label among multiple labels is removed , LABEL_DELETE is audited. But 
> when all labels are deleted or the last label is deleted , no LABEL_DELETE is 
> audited.
> Also , we see "One or more existing labels were removed successfully" popup 
> when there is no LABEL_DELETE audit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3601) No LABEL_DELETE event is audited when last label is deleted

2020-02-02 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3601:
--
Fix Version/s: 2.1.0

>  No LABEL_DELETE event is audited when last label is deleted
> 
>
> Key: ATLAS-3601
> URL: https://issues.apache.org/jira/browse/ATLAS-3601
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: ATLAS-3601.patch
>
>
> When 1 label among multiple labels is removed , LABEL_DELETE is audited. But 
> when all labels are deleted or the last label is deleted , no LABEL_DELETE is 
> audited.
> Also , we see "One or more existing labels were removed successfully" popup 
> when there is no LABEL_DELETE audit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72060: ATLAS-3601 No LABEL_DELETE event is audited when last label is deleted.

2020-02-02 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72060/#review219465
---


Ship it!




Ship It!

- Sarath Subramanian


On Jan. 30, 2020, 4:10 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72060/
> ---
> 
> (Updated Jan. 30, 2020, 4:10 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3601
> https://issues.apache.org/jira/browse/ATLAS-3601
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1) When 1 label among multiple labels is removed , LABEL_DELETE is audited. 
> But when all labels are deleted or the last label is deleted , no 
> LABEL_DELETE is audited.
> 2)When we add label and delete label from multiple label present, 
> simultaneously, Only LABEL_ADD is audited
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  f6b741b5e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  e427a59e7 
> 
> 
> Diff: https://reviews.apache.org/r/72060/diff/1/
> 
> 
> Testing
> ---
> 
> 1) Added one label - one audit for LABEL_ADD
> 2) Added multiple labels simultaneously - one audit for LABEL_ADD,with 
> multiple label names in Details
> 3) Deleted one of the label - one audit for LABEL_DELETE
> 4) Deleted multiple labels simultaneously - one audit for LABEL_DELETE,with 
> multiple label names in Details
> 5) deleled last label left - one audit for LABEL_DELETE
> 6) Added one label and deleted two labels simultaneously -two audits, one 
> audit for LABEL_ADD and second for LABEL_DELETE ,with multiple label names in 
> Details
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72060: ATLAS-3601 No LABEL_DELETE event is audited when last label is deleted.

2020-02-02 Thread Sarath Subramanian 


> On Jan. 30, 2020, 2:48 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
> > Line 161 (original), 161 (patched)
> > 
> >
> > how does adding a new field to hbase rowkey fix the issue? We are 
> > changing the rowkey format here. Will this be backward compatible? will 
> > older rows be retrieved using the new key format? Please review.
> 
> Pinal Shah wrote:
> **Issue-2: When we add label and delete label from multiple label 
> present, simultaneously, Only LABEL_ADD is audited**
> 
> *how does adding a new field to hbase rowkey fix the issue?*
> => As row_key(hbase) of both the audits were same, hbase couldnt insert 
> both the audits, for this we need unique row_key for each audit,so whenever 
> getKey() of AbstractStorageBasedAuditRepository class is called, it will 
> assure to return unique Key
> 
> https://github.com/apache/atlas/blob/master/repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java#L158
> 
> *We are changing the rowkey format here*
> => Yes old row_key = ::
>  new row_key = 
> :::   
> https://github.com/apache/atlas/blob/master/repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java#L158
> 
> *Will this be backward compatible? will older rows be retrieved using the 
> new key format?*
> => Yes it will be backward compatible and will retrieve older rows.
> => For retrieval, we make use of entityId and from the result, we extract 
> only entityId and timestamp from row_key
>below is the snippet.
>   String[] parts = key.split(FIELD_SEPARATOR);
> event.setEntityId(parts[0]);
> event.setTimestamp(Long.valueOf(parts[1]));
>
> https://github.com/apache/atlas/blob/master/repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java#L464

Thanks for the detailed explanation Pinal.


- Sarath


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72060/#review219442
---


On Jan. 30, 2020, 4:10 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72060/
> ---
> 
> (Updated Jan. 30, 2020, 4:10 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3601
> https://issues.apache.org/jira/browse/ATLAS-3601
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1) When 1 label among multiple labels is removed , LABEL_DELETE is audited. 
> But when all labels are deleted or the last label is deleted , no 
> LABEL_DELETE is audited.
> 2)When we add label and delete label from multiple label present, 
> simultaneously, Only LABEL_ADD is audited
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  f6b741b5e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  e427a59e7 
> 
> 
> Diff: https://reviews.apache.org/r/72060/diff/1/
> 
> 
> Testing
> ---
> 
> 1) Added one label - one audit for LABEL_ADD
> 2) Added multiple labels simultaneously - one audit for LABEL_ADD,with 
> multiple label names in Details
> 3) Deleted one of the label - one audit for LABEL_DELETE
> 4) Deleted multiple labels simultaneously - one audit for LABEL_DELETE,with 
> multiple label names in Details
> 5) deleled last label left - one audit for LABEL_DELETE
> 6) Added one label and deleted two labels simultaneously -two audits, one 
> audit for LABEL_ADD and second for LABEL_DELETE ,with multiple label names in 
> Details
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72046: ATLAS-3534 EntityREST changes to allow namespace attributes to be added, updated and deleted to an entity instance

2020-02-02 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72046/#review219463
---



In addition to the added REST APIs we need a REST API to 
setNamespaceAttributes, which the UI will be using to set namespace attributes. 
Server-side code should compare before and after state of namespace attributes 
and identify namespace attributes added/updated/removed and call appropriate 
internal methods for them.

Please make sure to validate the following cases:

```
1. Add Namespace attribute:
---
* ns2: attr5 = val5
* ns3: attr6 = val6

Before:
__
| ns1 | attr1 = val1 | ns1.attr1 = val1
| | attr2 = val2 | ns1.attr2 = val2
|-|--| ns2.attr3 = val3
| ns2 | attr3 = val3 | ns2.attr4 = val4
| | attr4 = val4 |
--

After:
___
| ns1 | attr1 = val1  |
| | attr2 = val2  |ns1.attr1 = val1
|-|---|ns1.attr2 = val2
| ns2 | attr3 = val3  |ns2.attr3 = val3
| | attr4 = val4  |ns2.attr4 = val4
| | attr5 = val5 +|ns2.attr5 = val5 +
|-|-- |ns3.attr6 = val6 +
| ns3 | attr6 = val6 +|
---


2. Remove Namespace attribute:
--
* ns1: attr2 = val2
* ns2: attr4 = val4

Before:
___
| ns1 | attr1 = val1  |
| | attr2 = val2 x|ns1.attr1 = val1
|-|---|ns1.attr2 = val2 x
| ns2 | attr3 = val3  |ns2.attr3 = val3
| | attr4 = val4 x|ns2.attr4 = val4 x
| | attr5 = val5  |ns2.attr5 = val5
|-|---|ns3.attr6 = val6
| ns3 | attr6 = val6  |
---

After:
__
| ns1 | attr1 = val1 |
|-|--|ns1.attr1 = val1
| ns2 | attr3 = val3 |ns2.attr3 = val3
| | attr5 = val5 |ns2.attr5 = val5
|-|--|ns3.attr6 = val6
| ns3 | attr6 = val6 |
--


3. Update Namespace attribute:
--
* ns2: attr3 = val333
* ns3: attr6 = val666

Before:
__
| ns1 | attr1 = val1 |
|-|--|ns1.attr1 = val1
| ns2 | attr3 = val3 |ns2.attr3 = val3
| | attr5 = val5 |ns2.attr5 = val5
|-|--|ns3.attr6 = val6
| ns3 | attr6 = val6 |
--

After:
__
| ns1 | attr1 = val1 |
|-|--|ns1.attr1 = val1
| ns2 | attr3 = val333 * |ns2.attr3 = val333 *
| | attr5 = val5 |ns2.attr5 = val5
|-|--|ns3.attr6 = val666 *
| ns3 | attr6 = val666 * |
--


4. Set Namespace Attribute:
---

Before: N1
__
| ns1 | attr1 = val1 | ns1.attr1 = val1
| | attr2 = val2 | ns1.attr2 = val2
|-|--| ns2.attr3 = val3
| ns2 | attr3 = val3 | ns2.attr4 = val4
| | attr4 = val4 |
--

After: N2

| ns1 | attr1 = val1   |
|-||ns1.attr1 = val1
| ns2 | attr3 = val333 |ns2.attr3 = val333
| | attr5 = val5   |ns2.attr5 = val5
|-||ns3.attr6 = val666
| ns3 | attr6 = val666 |



Added:
--
N2 - N1

Updated:

N1 = N2

Removed:

N1 - N2
```

- Sarath Subramanian


On Jan. 31, 2020, 4:40 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72046/
> ---
> 
> (Updated Jan. 31, 2020, 4:40 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3534
> https://issues.apache.org/jira/browse/ATLAS-3534
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Created separate endpoints to associate Namespace attributes to entity
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 0b28243 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 2054513 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 1b033b9 
>   intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java a141d4a 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 97e27d0 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
>  928c70d 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  25284e9 
>   
>

Re: Review Request 72046: ATLAS-3534 EntityREST changes to allow namespace attributes to be added, updated and deleted to an entity instance

2020-02-02 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72046/#review219459
---




intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java
Lines 133 (patched)


AtlasNamespaceType doesn't have any instance (compared to 
AtlasEntityType/AtlasStructType/AtlasEnumType/AtlasRelationshipType/AtlasClassificationType),
 hence following abstract methods should be empty/no-op:
 - createDefaultValue()
 - isValidValue()
 - areEqualValues()
 - getNormalizedValue()
 - validateValue()
 - isValidValueForUpdate()
 - getNormalizedValueForUpdate()
 - validateValueForUpdate()

Above methods in AtlasEntityType should be updated to handle namespace 
attributes. Please review and update.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
Lines 824 (patched)


- checkIfNamespaceAttributesExist() is called only within 
AtlasEntityStoreV2. Consider marking this method as private
- given checkIfNamespaceAttributesExist() returns true if the vertex has at 
least one namespace attribute, consider renaming to 
containsAnyNamespaceAttribute()



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
Lines 915 (patched)


checkIfNamespaceAttributesExist() returns true on presence of first 
namespace-attribute, which may not be good here. Consider adding method that 
checks for presence of all given namespace-attributes - 
containsAllNamespaceAttributes(entityVertex, namespaceAttributes)



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
Lines 1449 (patched)


Consider following simper validation:

  for (String nsName : entityNamespaces.keySet()) {
List nsAttributes = 
entityType.getNamespaceAttributes(nsName);

if (nsAttributes == null) {
  messages.add(nsName + ": invalid namespace for entity type " + 
entityType.getTypeName());

  continue;
}

Map entityNsAttributes = entityNamespaces.get(nsName);

for (AtlasNamespaceAttribute nsAttribute : nsAttributes) {
  AtlasType attrType  = nsAttribute.getAttributeType();
  StringattrName  = nsAttribute.getName();
  ObjectattrValue = entityNsAttributes.get(attrName);
  StringfieldName = entityType.getTypeName() + "." + nsName + "." + 
attrName;
  
  if (attrValue != null) {
attrType.validateValue(attrValue, fieldName, messages);
  } else if (!nsAttribute.getAttributeDef().getIsOptional()) {
messages.add(fieldName + ": mandatory namespace attribute value 
missing in type " + entityType.getTypeName());
  }
}
  }



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Line 617 (original), 620 (patched)


When 'attributes' contains namespace-attributes (named as 
"namespaceName:attributeName"), returned entity should have these 
namespace-attributes populated as well.

This update can be taken up in a subsequent patch, once basic-search is 
updated to support namespace attributes.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Line 694 (original), 697 (patched)


When 'attributes' contains namespace-attributes (named as 
"namespaceName:attributeName"?), returned entity should have these 
namespace-attributes populated as well.

This update can be taken up in a subsequent patch, once basic-search is 
updated to support namespace attributes.


- Madhan Neethiraj


On Jan. 31, 2020, 12:40 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72046/
> ---
> 
> (Updated Jan. 31, 2020, 12:40 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3534
> https://issues.apache.org/jira/browse/ATLAS-3534
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Created separate endpoints to associate Namespace attributes to entity
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 0b28243 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 2054513 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 1b033b9 
>