[jira] [Updated] (ATLAS-4294) Format long description of Glossary term and category UI

2021-05-20 Thread Nixon Rodrigues (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nixon Rodrigues updated ATLAS-4294:
---
Summary: Format long description of Glossary term and category UI  (was: 
Format long description of Glossary term and category)

> Format long description of Glossary term and category UI
> 
>
> Key: ATLAS-4294
> URL: https://issues.apache.org/jira/browse/ATLAS-4294
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Prasad P. Pawar
>Priority: Major
>
> Currently the long description is render as long string  and it all appears 
> as a single text string
> It is hard to read multiple lines.
> Makes to support newline character in rendering long description
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4294) Atlas UI : Format long description of Glossary term and category UI

2021-05-20 Thread Nixon Rodrigues (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nixon Rodrigues updated ATLAS-4294:
---
Summary: Atlas UI : Format long description of Glossary term and category 
UI  (was: Format long description of Glossary term and category UI)

> Atlas UI : Format long description of Glossary term and category UI
> ---
>
> Key: ATLAS-4294
> URL: https://issues.apache.org/jira/browse/ATLAS-4294
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Prasad P. Pawar
>Priority: Major
>
> Currently the long description is render as long string  and it all appears 
> as a single text string
> It is hard to read multiple lines.
> Makes to support newline character in rendering long description
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (ATLAS-4294) Format long description of Glossary term and category

2021-05-20 Thread Prasad P. Pawar (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad P. Pawar reassigned ATLAS-4294:
--

Assignee: Prasad P. Pawar

> Format long description of Glossary term and category
> -
>
> Key: ATLAS-4294
> URL: https://issues.apache.org/jira/browse/ATLAS-4294
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Prasad P. Pawar
>Priority: Major
>
> Currently the long description is render as long string  and it all appears 
> as a single text string
> It is hard to read multiple lines.
> Makes to support newline character in rendering long description
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (ATLAS-4292) [Atlas: Debug Metrics] com.sun.jersey.api.MessageException thrown while fetching debug metrics via browser

2021-05-20 Thread Mayank Jain (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Jain reassigned ATLAS-4292:
--

Assignee: Mayank Jain

> [Atlas: Debug Metrics] com.sun.jersey.api.MessageException thrown while 
> fetching debug metrics via browser
> --
>
> Key: ATLAS-4292
> URL: https://issues.apache.org/jira/browse/ATLAS-4292
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Mayank Jain
>Priority: Major
> Attachments: Screenshot 2021-05-20 at 7.44.34 PM.png, Screenshot 
> 2021-05-20 at 8.03.07 PM.png
>
>
> While fetching the data via browser, the following exception is thrown
> {code:java}
> 2021-05-20 13:47:32,150 INFO  - 
> [etp522553046-44:HTTP:GET/api/atlas/admin/metrics] ~ Request from 
> authenticated user: HTTP, URL=/api/atlas/admin/metrics 
> (AtlasAuthenticationFilter$KerberosFilterChainWrapper:739)
> 2021-05-20 13:47:50,657 ERROR - [etp522553046-269 - 
> 41f0e041-f379-4389-b998-62bb43cafd88:] ~ Error handling a request: 
> 58173f6d7e3447db (ExceptionMapperUtil:32)at 
> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:284)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at 
> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452)
> at 
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
> at 
> org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
> at 
> org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:106)
> at 
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at 
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at 
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at 
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.apache.atlas.web.filters.AtlasCSRFPreventionFilter$ServletFilterHttpInteraction.proceed(AtlasCSRFPreventionFilter.java:235)
> at 
> org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.handleHttpInteraction(AtlasCSRFPreventionFilter.java:177)
> at 
> org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.doFilter(AtlasCSRFPreventionFilter.java:190)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.apache.atlas.web.filters.AtlasAuthenticationFilter.doFilter(AtlasAuthenticationFilter.java:358)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
> at 
>

[jira] [Created] (ATLAS-4294) Format long description of Glossary term and category

2021-05-20 Thread Nixon Rodrigues (Jira) 
Nixon Rodrigues created ATLAS-4294:
--

 Summary: Format long description of Glossary term and category
 Key: ATLAS-4294
 URL: https://issues.apache.org/jira/browse/ATLAS-4294
 Project: Atlas
  Issue Type: Improvement
Reporter: Nixon Rodrigues


Currently the long description is render as long string  and it all appears as 
a single text string

It is hard to read multiple lines.

Makes to support newline character in rendering long description

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73329: Correlating Deleted Entities with Lineage

2021-05-20 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/#review223021
---


Ship it!




Ship It!

- Sarath Subramanian


On May 20, 2021, 3:18 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73329/
> ---
> 
> (Updated May 20, 2021, 3:18 p.m.)
> 
> 
> Review request for atlas, Radhika Kundam and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4152
> https://issues.apache.org/jira/browse/ATLAS-4152
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided 
> the poperty is enabled).
> 
> When Spooling enabled the order of messages can potentially change. The 
> notification messages coming from HS2 and HMS may not be in the same order as 
> when they arrived with direct notification.
> 
> Problem: 
> Consider the sequence of arriving messages:
> 
> This is the sequence of messages for Entity 1 (C = create, U = update, D = 
> delete, L?x = Lineage of type 'x')
> No problem: C1, U1, L1x, L1y, D1
> Problem: C1, U1, D1, L1x, L1y
> 
> This implementation attempts to handle ths problem mentioned above. If the 
> above case is not handled, it will end up creating shell entities, since 
> deleted entities are not looked up as part of entity creation.
> 
> **Approach**
> Used bounded stream approach where an incoming stream of messages is bounded 
> with an indicator that it originates from spool. This helps makes localized 
> decisions on the incoming stream of messages.
> 
> High-level approach:
> - Messages when written to the spool are tagged with a timestamp.
> - Deleted entities are maintained in a cache.
> - Lineage-only message are checked if they refer to a deleted entity.
> - If they refer to deleted entity, they are stitched to the one present in 
> the cache only if it falls within the threshold.
> - Using step-climbing approach for locating right entity to stitch lineage to.
> 
> New: _EntityCorrelationsManager_: Uses message timestamp and cached entity 
> qualifiedName-GUID map.
> Modifed: _NotificationHookConsumer_ Uses the new class.
> New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted 
> entities.
> Modified: _SpoolConfiguration_: Added new configuration to pause message 
> sending after destination is available: 
> _atlas.hook.spool.pause.before.send.sec_.
> In-memory lookup approach changed to persistent loookup.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java ffcec9743 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  810ba97c9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> f7d9668ec 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> 22bd79fdf 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 3d1b3ccf1 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
>  3264e264c 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  edd8ed931 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  2d7d19595 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java 
> 22242c933 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
>  a9a3a78cc 
>   notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java 
> 2cacaaadc 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  d7e4959f7 
>   
> notification/src/test/java/org/apache/atlas/notification/spool/AtlasFileSpoolTest.java
>  167efbecc 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  cc727c6ba 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/EntityCorrelationStore.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
>  0a9470815 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java
>  PRE-CREATION 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
>  PRE-CREATION 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  84cc8d813 
>   
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java
>  89568e236 
>   
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDbDDLPreprocessor.java
>  PRE-CREATION 
>   
>

Re: Review Request 73329: Correlating Deleted Entities with Lineage

2021-05-20 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/
---

(Updated May 20, 2021, 10:18 p.m.)


Review request for atlas, Radhika Kundam and Sarath Subramanian.


Changes
---

Updates include: Addressed review comments.


Bugs: ATLAS-4152
https://issues.apache.org/jira/browse/ATLAS-4152


Repository: atlas


Description
---

**Background**
As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided the 
poperty is enabled).

When Spooling enabled the order of messages can potentially change. The 
notification messages coming from HS2 and HMS may not be in the same order as 
when they arrived with direct notification.

Problem: 
Consider the sequence of arriving messages:

This is the sequence of messages for Entity 1 (C = create, U = update, D = 
delete, L?x = Lineage of type 'x')
No problem: C1, U1, L1x, L1y, D1
Problem: C1, U1, D1, L1x, L1y

This implementation attempts to handle ths problem mentioned above. If the 
above case is not handled, it will end up creating shell entities, since 
deleted entities are not looked up as part of entity creation.

**Approach**
Used bounded stream approach where an incoming stream of messages is bounded 
with an indicator that it originates from spool. This helps makes localized 
decisions on the incoming stream of messages.

High-level approach:
- Messages when written to the spool are tagged with a timestamp.
- Deleted entities are maintained in a cache.
- Lineage-only message are checked if they refer to a deleted entity.
- If they refer to deleted entity, they are stitched to the one present in the 
cache only if it falls within the threshold.
- Using step-climbing approach for locating right entity to stitch lineage to.

New: _EntityCorrelationsManager_: Uses message timestamp and cached entity 
qualifiedName-GUID map.
Modifed: _NotificationHookConsumer_ Uses the new class.
New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted 
entities.
Modified: _SpoolConfiguration_: Added new configuration to pause message 
sending after destination is available: 
_atlas.hook.spool.pause.before.send.sec_.
In-memory lookup approach changed to persistent loookup.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java ffcec9743 
  
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
 810ba97c9 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
f7d9668ec 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
22bd79fdf 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
3d1b3ccf1 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
 3264e264c 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 edd8ed931 
  
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
 2d7d19595 
  notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java 
22242c933 
  
notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
 a9a3a78cc 
  notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java 
2cacaaadc 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 d7e4959f7 
  
notification/src/test/java/org/apache/atlas/notification/spool/AtlasFileSpoolTest.java
 167efbecc 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 cc727c6ba 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/EntityCorrelationStore.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
 0a9470815 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 84cc8d813 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java
 89568e236 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDbDDLPreprocessor.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
 86e3384ee 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveTableDDLPreprocessor.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java
 608b4a304 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
 65e8b5001 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
 15a19005e 


Diff:

[jira] [Updated] (ATLAS-4285) AtlasTasks: Multiple tag propagation tasks running concurrently, task is complete but propagation is not complete

2021-05-20 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4285:
--
Component/s:  atlas-core

> AtlasTasks: Multiple tag propagation tasks running concurrently, task is 
> complete but propagation is not complete
> -
>
> Key: ATLAS-4285
> URL: https://issues.apache.org/jira/browse/ATLAS-4285
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: 
> ATLAS-4285-Multiple-propagations-with-intersecting-l.patch
>
>
> Created a 500 level linear lineage . (table1 ---> table2 ---> table3 ---> 
> .. ---> table500)
> Added tag1 to table1 
> Added tag2 to table2
> Added tag3 to table3 
> 3 tasks are created.
> task2 got completed and tag2 is associated only to table2 and not propagated 
> till table500.
> After sometime all tasks are completed , but propagation didn't happen



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4285) AtlasTasks: Multiple tag propagation tasks running concurrently, task is complete but propagation is not complete

2021-05-20 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4285:
--
Fix Version/s: 2.2.0
   3.0.0

> AtlasTasks: Multiple tag propagation tasks running concurrently, task is 
> complete but propagation is not complete
> -
>
> Key: ATLAS-4285
> URL: https://issues.apache.org/jira/browse/ATLAS-4285
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: 
> ATLAS-4285-Multiple-propagations-with-intersecting-l.patch
>
>
> Created a 500 level linear lineage . (table1 ---> table2 ---> table3 ---> 
> .. ---> table500)
> Added tag1 to table1 
> Added tag2 to table2
> Added tag3 to table3 
> 3 tasks are created.
> task2 got completed and tag2 is associated only to table2 and not propagated 
> till table500.
> After sometime all tasks are completed , but propagation didn't happen



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4285) AtlasTasks: Multiple tag propagation tasks running concurrently, task is complete but propagation is not complete

2021-05-20 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4285:
--
Labels: deferred-actions tagpropagation  (was: )

> AtlasTasks: Multiple tag propagation tasks running concurrently, task is 
> complete but propagation is not complete
> -
>
> Key: ATLAS-4285
> URL: https://issues.apache.org/jira/browse/ATLAS-4285
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
>  Labels: deferred-actions, tagpropagation
> Fix For: 3.0.0, 2.2.0
>
> Attachments: 
> ATLAS-4285-Multiple-propagations-with-intersecting-l.patch
>
>
> Created a 500 level linear lineage . (table1 ---> table2 ---> table3 ---> 
> .. ---> table500)
> Added tag1 to table1 
> Added tag2 to table2
> Added tag3 to table3 
> 3 tasks are created.
> task2 got completed and tag2 is associated only to table2 and not propagated 
> till table500.
> After sometime all tasks are completed , but propagation didn't happen



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4285) AtlasTasks: Multiple tag propagation tasks running concurrently, task is complete but propagation is not complete

2021-05-20 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/ATLAS-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348818#comment-17348818
 ] 

ASF subversion and git services commented on ATLAS-4285:


Commit 3016ba6fb16f2f4b598eefc48e97ac2c50f16a84 in atlas's branch 
refs/heads/branch-2.0 from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=3016ba6 ]

ATLAS-4285: Multiple propagations with intersecting lineage.

Signed-off-by: Sarath Subramanian 
(cherry picked from commit aba97b35393f1732eb30858e69fd5f489634afdc)


> AtlasTasks: Multiple tag propagation tasks running concurrently, task is 
> complete but propagation is not complete
> -
>
> Key: ATLAS-4285
> URL: https://issues.apache.org/jira/browse/ATLAS-4285
> Project: Atlas
>  Issue Type: Bug
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: 
> ATLAS-4285-Multiple-propagations-with-intersecting-l.patch
>
>
> Created a 500 level linear lineage . (table1 ---> table2 ---> table3 ---> 
> .. ---> table500)
> Added tag1 to table1 
> Added tag2 to table2
> Added tag3 to table3 
> 3 tasks are created.
> task2 got completed and tag2 is associated only to table2 and not propagated 
> till table500.
> After sometime all tasks are completed , but propagation didn't happen



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4285) AtlasTasks: Multiple tag propagation tasks running concurrently, task is complete but propagation is not complete

2021-05-20 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/ATLAS-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348817#comment-17348817
 ] 

ASF subversion and git services commented on ATLAS-4285:


Commit aba97b35393f1732eb30858e69fd5f489634afdc in atlas's branch 
refs/heads/master from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=aba97b3 ]

ATLAS-4285: Multiple propagations with intersecting lineage.

Signed-off-by: Sarath Subramanian 


> AtlasTasks: Multiple tag propagation tasks running concurrently, task is 
> complete but propagation is not complete
> -
>
> Key: ATLAS-4285
> URL: https://issues.apache.org/jira/browse/ATLAS-4285
> Project: Atlas
>  Issue Type: Bug
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: 
> ATLAS-4285-Multiple-propagations-with-intersecting-l.patch
>
>
> Created a 500 level linear lineage . (table1 ---> table2 ---> table3 ---> 
> .. ---> table500)
> Added tag1 to table1 
> Added tag2 to table2
> Added tag3 to table3 
> 3 tasks are created.
> task2 got completed and tag2 is associated only to table2 and not propagated 
> till table500.
> After sometime all tasks are completed , but propagation didn't happen



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4285) AtlasTasks: Multiple tag propagation tasks running concurrently, task is complete but propagation is not complete

2021-05-20 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4285:
--
Affects Version/s: 2.1.0

> AtlasTasks: Multiple tag propagation tasks running concurrently, task is 
> complete but propagation is not complete
> -
>
> Key: ATLAS-4285
> URL: https://issues.apache.org/jira/browse/ATLAS-4285
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: 
> ATLAS-4285-Multiple-propagations-with-intersecting-l.patch
>
>
> Created a 500 level linear lineage . (table1 ---> table2 ---> table3 ---> 
> .. ---> table500)
> Added tag1 to table1 
> Added tag2 to table2
> Added tag3 to table3 
> 3 tasks are created.
> task2 got completed and tag2 is associated only to table2 and not propagated 
> till table500.
> After sometime all tasks are completed , but propagation didn't happen



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73356: Improve Tag Propagation in Entities Involved in Several Lineages

2021-05-20 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73356/#review223020
---


Ship it!




Ship It!

- Sarath Subramanian


On May 14, 2021, 9:48 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73356/
> ---
> 
> (Updated May 14, 2021, 9:48 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Nikhil Bonte, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4285
> https://issues.apache.org/jira/browse/ATLAS-4285
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Steps to Duplicate**
> _Setup_:
> - Create a linear lineage using chained CTAS queries via Hive or REST APIs 
> E.g. 
> ```
> create view abcd_vw_1 as select * from abc;
> create view abcd_vw_2 as select * from abcd_vw_1;
> create view abcd_vw_3 as select * from abcd_vw_2;
> 
> .
> .
> .
> 
> ```
> This should result in a long lineage.
> - Create 3 classifications, say C1, C2, C3.
> - Assign C1 to abcd_vw_1, C2 to abcd_vw_2 and C3 to abcd_vw_3.
> 
> Expected result: abcd_vw_n should have all the 3 classifications propagated.
> 
> **Aproach**
> Modify _propagateClassification_ to lock all the impacted vertices. Requests 
> attempting to modify the same entity will thus be blocked.
> 
> **Additional Information**
> - Discovered object creation sequence change for _TaskMangement_ and 
> _TaskFactoryRegistry_. This causes pending tasks not to be scheduled.
> - Removed _ClassificationPropagationTasks.Update_ since it is not used.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  7984a34a4 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
>  8a81dc952 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
>  f86cbc722 
>   repository/src/main/java/org/apache/atlas/tasks/TaskFactoryRegistry.java 
> 38f2cc9cd 
>   repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 
> 2756504da 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationWithTasksTest.java
>  84aefc9ef 
> 
> 
> Diff: https://reviews.apache.org/r/73356/diff/1/
> 
> 
> Testing
> ---
> 
> **Manual tests**
> Used the scenario described above. See the attached file used to generate 
> lineage.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73329: Correlating Deleted Entities with Lineage

2021-05-20 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/#review223017
---




common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 138 (patched)


"deletedMessageTimestamp" => entityDeletedTimestamp

DELETED_MESSAGE_TIMESTAMP_PROPERTY_KEY => 
ENTITY_DELETED_TIMESTAMP_PROPERTY_KEY



notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java
Lines 216 (patched)


consider adding a comment - explaining the reason for adding sleep for hs2 
messages in spool



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
Lines 377 (patched)


reorder the garph query conditions for better perf:

graph.query().has(STATE_PROPERTY_KEY, Status.DELETED.name())
 .has(Constants.DELETED_MESSAGE_TIMESTAMP_PROPERTY_KEY, 
GREATER_THAN, timestamp)
 .has(Constants.QUALIFIED_NAME, qualifiedName)
 .orderBy(Constants.DELETED_MESSAGE_TIMESTAMP_PROPERTY_KEY, 
ASC);



webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDbDDLPreprocessor.java
Lines 38 (patched)


dbRelationship => dbObject



webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveTableDDLPreprocessor.java
Lines 38 (patched)


tableObject


- Sarath Subramanian


On May 19, 2021, 9 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73329/
> ---
> 
> (Updated May 19, 2021, 9 p.m.)
> 
> 
> Review request for atlas, Radhika Kundam and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4152
> https://issues.apache.org/jira/browse/ATLAS-4152
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided 
> the poperty is enabled).
> 
> When Spooling enabled the order of messages can potentially change. The 
> notification messages coming from HS2 and HMS may not be in the same order as 
> when they arrived with direct notification.
> 
> Problem: 
> Consider the sequence of arriving messages:
> 
> This is the sequence of messages for Entity 1 (C = create, U = update, D = 
> delete, L?x = Lineage of type 'x')
> No problem: C1, U1, L1x, L1y, D1
> Problem: C1, U1, D1, L1x, L1y
> 
> This implementation attempts to handle ths problem mentioned above. If the 
> above case is not handled, it will end up creating shell entities, since 
> deleted entities are not looked up as part of entity creation.
> 
> **Approach**
> Used bounded stream approach where an incoming stream of messages is bounded 
> with an indicator that it originates from spool. This helps makes localized 
> decisions on the incoming stream of messages.
> 
> High-level approach:
> - Messages when written to the spool are tagged with a timestamp.
> - Deleted entities are maintained in a cache.
> - Lineage-only message are checked if they refer to a deleted entity.
> - If they refer to deleted entity, they are stitched to the one present in 
> the cache only if it falls within the threshold.
> - Using step-climbing approach for locating right entity to stitch lineage to.
> 
> New: _EntityCorrelationsManager_: Uses message timestamp and cached entity 
> qualifiedName-GUID map.
> Modifed: _NotificationHookConsumer_ Uses the new class.
> New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted 
> entities.
> Modified: _SpoolConfiguration_: Added new configuration to pause message 
> sending after destination is available: 
> _atlas.hook.spool.pause.before.send.sec_.
> In-memory lookup approach changed to persistent loookup.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java ffcec9743 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  810ba97c9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> f7d9668ec 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> 22bd79fdf 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 3d1b3ccf1 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
>  3264e264c 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  edd8ed931 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  2d7d19595 
>   
>

[jira] [Updated] (ATLAS-4293) Table changes are not propagated to Atlas in Create Table with External Storage Handler and few scenarios of Alter table

2021-05-20 Thread Radhika Kundam (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radhika Kundam updated ATLAS-4293:
--
Description: 
*Scenario-1:* Create external table pointing to external storage handler 
hbase/kafka

In this case there is no entity created for the external hbase path, though 
there is atlas entity for hbase table.Also no lineage present on atlas for this 
table to external path.

Query for Hive Table with external HBase storage handler:

create table default.tbl29_hbs(key int, value string)
 STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
 WITH SERDEPROPERTIES (
 "hbase.columns.mapping" = ":key,cf:name",
 "hbase.table.name" = "default:hbs_tbl29"
 );

 

Below are the Alter table scenarios which are not propagated to Atlas

*Scenario-2:* SET SERDE

 
{code:java}
CREATE TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 (a int, b int, c int, 
d int, e int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 SET SERDE 
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
 
Assert Error:
 
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 61 sec <<< FAILURE!
 java.lang.AssertionError: The following asserts failed:
 serdeInfo serializationLib mismatch expected 
[org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe] but found 
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at 
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
 

*Scenario-3:* SET FILEFORMAT TEXTFILE
{code:java}
CREATE TABLE db2k9eooxraj5.text_table_k9eooxraj5 (a int, b int, c int, d int, e 
int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.text_table_k9eooxraj5 SET FILEFORMAT TEXTFILE
 Assert Error:
 
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 60.788 sec <<< FAILURE!
 java.lang.AssertionError: The following asserts failed:
 outputFormat mismatch expected 
[org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat] but found 
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat 
mismatch expected [org.apache.hadoop.mapred.TextInputFormat] but found 
[org.apache.hadoop.mapred.SequenceFileInputFormat]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at 
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
 

*Scenario-4:* SET FILEFORMAT SEQUENCEFILE
{code:java}
CREATE TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 (a int, b int, c int, d 
int, e int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 SET FILEFORMAT SEQUENCEFILE
 
Assert Error:
 
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 60.796 sec <<< FAILURE!
 java.lang.AssertionError: The following asserts failed:
 outputFormat mismatch expected 
[org.apache.hadoop.mapred.SequenceFileOutputFormat] but found 
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at 
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-5:* SET FILEFORMAT PARQUET
{code:java}
CREATE TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 (a int, b int, c int, d 
int, e int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 SET FILEFORMAT PARQUET
 
Assert Error:
 
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 60.803 sec <<< FAILURE!
 java.lang.AssertionError: The following asserts failed:
 serdeInfo serializationLib mismatch expected 
[org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe] but found 
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe], outputFormat mismatch 
expected [org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat] but 
found [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat 
mismatch expected 
[org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat] but found 
[org.apache.hadoop.mapred.SequenceFileInputFormat]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at 
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-6:* ALTER TABLE using CLUSTERED BY
{code:java}
create table db2k9eooxraj5.transaction_table_k9eooxraj5 (EmployeeID 
Int,FirstName

[jira] [Updated] (ATLAS-4293) Table changes are not propagated to Atlas in Create Table with External Storage Handler and few scenarios of Alter table

2021-05-20 Thread Radhika Kundam (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radhika Kundam updated ATLAS-4293:
--
Description: 
*Scenario-1:* Create external table pointing to external storage handler 
hbase/kafka

In this case there is no entity created for the external hbase path, though 
there is atlas entity for hbase table.Also no lineage present on atlas for this 
table to external path.

Query for Hive Table with external HBase storage handler:
{code:java}
create table default.tbl29_hbs(key int, value string)
 STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
 WITH SERDEPROPERTIES (
 "hbase.columns.mapping" = ":key,cf:name",
 "hbase.table.name" = "default:hbs_tbl29"
 );{code}
 

Below are the Alter table scenarios which are not propagated to Atlas

*Scenario-2:* SET SERDE
{code:java}
CREATE TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 (a int, b int, c int, 
d int, e int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 SET SERDE 
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
 
Assert Error:
 
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 61 sec <<< FAILURE!
 java.lang.AssertionError: The following asserts failed:
 serdeInfo serializationLib mismatch expected 
[org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe] but found 
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at 
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
 

*Scenario-3:* SET FILEFORMAT TEXTFILE
{code:java}
CREATE TABLE db2k9eooxraj5.text_table_k9eooxraj5 (a int, b int, c int, d int, e 
int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.text_table_k9eooxraj5 SET FILEFORMAT TEXTFILE
 Assert Error:
 
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 60.788 sec <<< FAILURE!
 java.lang.AssertionError: The following asserts failed:
 outputFormat mismatch expected 
[org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat] but found 
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat 
mismatch expected [org.apache.hadoop.mapred.TextInputFormat] but found 
[org.apache.hadoop.mapred.SequenceFileInputFormat]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at 
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
 

*Scenario-4:* SET FILEFORMAT SEQUENCEFILE
{code:java}
CREATE TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 (a int, b int, c int, d 
int, e int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 SET FILEFORMAT SEQUENCEFILE
 
Assert Error:
 
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 60.796 sec <<< FAILURE!
 java.lang.AssertionError: The following asserts failed:
 outputFormat mismatch expected 
[org.apache.hadoop.mapred.SequenceFileOutputFormat] but found 
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at 
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-5:* SET FILEFORMAT PARQUET
{code:java}
CREATE TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 (a int, b int, c int, d 
int, e int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 SET FILEFORMAT PARQUET
 
Assert Error:
 
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 60.803 sec <<< FAILURE!
 java.lang.AssertionError: The following asserts failed:
 serdeInfo serializationLib mismatch expected 
[org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe] but found 
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe], outputFormat mismatch 
expected [org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat] but 
found [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat 
mismatch expected 
[org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat] but found 
[org.apache.hadoop.mapred.SequenceFileInputFormat]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at 
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-6:* ALTER TABLE using CLUSTERED BY
{code:java}
create table db2k9eooxraj5.transaction_table_k9eooxraj5 (EmployeeID

[jira] [Created] (ATLAS-4293) Table changes are not propagated to Atlas in Create Table with External Storage Handler and few scenarios of Alter table

2021-05-20 Thread Radhika Kundam (Jira) 
Radhika Kundam created ATLAS-4293:
-

 Summary: Table changes are not propagated to Atlas in Create Table 
with External Storage Handler and few scenarios of Alter table
 Key: ATLAS-4293
 URL: https://issues.apache.org/jira/browse/ATLAS-4293
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 2.1.0
Reporter: Radhika Kundam
Assignee: Radhika Kundam


*Scenario-1:* Create Hive Table with external storage handler

In this case there is no entity created for the external hbase path, though 
there is atlas entity for hbase table.Also no lineage present on atlas for this 
table to external path.

Query for Hive Table with external storage handler:

*create table default.tbl29_hbs(key int, value string)*
*STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'*
*WITH SERDEPROPERTIES (*
*"hbase.columns.mapping" = ":key,cf:name",*
*"hbase.table.name" = "default:hbs_tbl29"*
*);*

Below are the Alter table scenarios which are not propagated to Atlas

*Scenario-2:* SET SERDE

CREATE TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 (a int, b int, c int, 
d int, e int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 SET SERDE 
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
*Assert Error:*
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 61 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
serdeInfo serializationLib mismatch expected 
[org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe] but found 
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at 
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)

*Scenario-3:* SET FILEFORMAT TEXTFILE
CREATE TABLE db2k9eooxraj5.text_table_k9eooxraj5 (a int, b int, c int, d int, e 
int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
alter table db2k9eooxraj5.text_table_k9eooxraj5 SET FILEFORMAT TEXTFILE
*Assert Error:*
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 60.788 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
outputFormat mismatch expected 
[org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat] but found 
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat 
mismatch expected [org.apache.hadoop.mapred.TextInputFormat] but found 
[org.apache.hadoop.mapred.SequenceFileInputFormat]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at 
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)

*Scenario-4:* SET FILEFORMAT SEQUENCEFILE

CREATE TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 (a int, b int, c int, d 
int, e int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
alter table db2k9eooxraj5.sequence_table_k9eooxraj5 SET FILEFORMAT SEQUENCEFILE
*Assert Error:*
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 60.796 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
outputFormat mismatch expected 
[org.apache.hadoop.mapred.SequenceFileOutputFormat] but found 
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at 
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)

*Scenario-5:* SET FILEFORMAT PARQUET
CREATE TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 (a int, b int, c int, d 
int, e int, f int) ROW FORMAT 
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
alter table db2k9eooxraj5.parquet_table_k9eooxraj5 SET FILEFORMAT PARQUET
*Assert Error:*
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
 Time elapsed: 60.803 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
serdeInfo serializationLib mismatch expected 
[org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe] but found 
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe], outputFormat mismatch 
expected [org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat] but 
found [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat 
mismatch expected 
[org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat] but found 
[org.apache.hadoop.mapred.SequenceFileInputFormat]
 at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
 at

[jira] [Created] (ATLAS-4292) [Atlas: Debug Metrics] com.sun.jersey.api.MessageException thrown while fetching debug metrics via browser

2021-05-20 Thread Dharshana M Krishnamoorthy (Jira) 
Dharshana M Krishnamoorthy created ATLAS-4292:
-

 Summary: [Atlas: Debug Metrics] 
com.sun.jersey.api.MessageException thrown while fetching debug metrics via 
browser
 Key: ATLAS-4292
 URL: https://issues.apache.org/jira/browse/ATLAS-4292
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Reporter: Dharshana M Krishnamoorthy
 Attachments: Screenshot 2021-05-20 at 7.44.34 PM.png, Screenshot 
2021-05-20 at 8.03.07 PM.png

While fetching the data via browser, the following exception is thrown
{code:java}
2021-05-20 13:47:32,150 INFO  - 
[etp522553046-44:HTTP:GET/api/atlas/admin/metrics] ~ Request from authenticated 
user: HTTP, URL=/api/atlas/admin/metrics 
(AtlasAuthenticationFilter$KerberosFilterChainWrapper:739)
2021-05-20 13:47:50,657 ERROR - [etp522553046-269 - 
41f0e041-f379-4389-b998-62bb43cafd88:] ~ Error handling a request: 
58173f6d7e3447db (ExceptionMapperUtil:32)at 
com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:284)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at 
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452)
at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at 
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at 
org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:106)
at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at 
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at 
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at 
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at 
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at 
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at 
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter$ServletFilterHttpInteraction.proceed(AtlasCSRFPreventionFilter.java:235)
at 
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.handleHttpInteraction(AtlasCSRFPreventionFilter.java:177)
at 
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.doFilter(AtlasCSRFPreventionFilter.java:190)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at 
org.apache.atlas.web.filters.AtlasAuthenticationFilter.doFilter(AtlasAuthenticationFilter.java:358)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at 
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at 
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at 
org.apache.atlas.web.filters.StaleTransactionCleanupFilter.doFilter(StaleTransactionCleanupFilter.java:55)
at

[jira] [Created] (ATLAS-4291) [Atlas: Debug Metrics] The Average time calculated for some endpoints/methods are incorrect

2021-05-20 Thread Dharshana M Krishnamoorthy (Jira) 
Dharshana M Krishnamoorthy created ATLAS-4291:
-

 Summary: [Atlas: Debug Metrics] The Average time calculated for 
some endpoints/methods are incorrect
 Key: ATLAS-4291
 URL: https://issues.apache.org/jira/browse/ATLAS-4291
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Reporter: Dharshana M Krishnamoorthy
 Attachments: Screenshot 2021-05-20 at 6.58.22 PM.png

The average time calculated for some of the requests are incorrect !Screenshot 
2021-05-20 at 6.58.22 PM.png|width=461,height=223!

*Repro steps:*

Enable debug metrics by setting *atlas.debug.metrics.enabled=true*

Perform some operations create/delete glossary/entity or any other operation

Open the REST API metrics page

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)