[jira] [Created] (ATLAS-4449) atlas版本2.0编译报依赖org.restlet错误

2021-10-10 Thread cjjxfli (Jira) 
cjjxfli created ATLAS-4449:
--

 Summary: atlas版本2.0编译报依赖org.restlet错误
 Key: ATLAS-4449
 URL: https://issues.apache.org/jira/browse/ATLAS-4449
 Project: Atlas
  Issue Type: Bug
  Components: atlas-intg
 Environment: centos7
Reporter: cjjxfli


[ERROR] Failed to execute goal on project atlas-testtools: Could not resolve 
dependencies for project org.apache.atlas:atlas-testtools:jar:2.3.0-SNAPSHOT: 
The following artifacts could not be resolved: 
org.restlet.jee:org.restlet:jar:2.4.3, 
org.restlet.jee:org.restlet.ext.servlet:jar:2.4.3: Could not find artifact 
org.restlet.jee:org.restlet:jar:2.4.3 in alimaven 
(http://maven.aliyun.com/nexus/content/groups/public/) -> [Help 1]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73589: Indexing Monitor and Recovery

2021-10-10 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73589/#review223594
---


Ship it!




Ship It!

- Sarath Subramanian


On Sept. 21, 2021, 8:40 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73589/
> ---
> 
> (Updated Sept. 21, 2021, 8:40 a.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4408
> https://issues.apache.org/jira/browse/ATLAS-4408
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This implementation builds up on Radhika's earlier implementation. 
> https://reviews.apache.org/r/73555/
> 
> Indexing
> 
> atlas.graph.index.status.check.frequency: Frequence of checking Solr for 
> health status.
> atlas.graph.index.recovery.start.time: User can set recovery time in UTC. 
> E.g. 
> ```
> atlas.graph.index.recovery.start.time=2014-12-11T02:35:58.440Z
> ```
> 
> **Implementation**
> New: _updateGlobalConfiguration_: Updating global configuration variables.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
>  9960d8909 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  7e3b2f440 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
>  11267c438 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphIndexClient.java
>  ef42dbd3d 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  1cc7f8b55 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java 682206dad 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  ddfb0089e 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/RecoveryInfoManagementTest.java
>  PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  ba8f0886c 
> 
> 
> Diff: https://reviews.apache.org/r/73589/diff/2/
> 
> 
> Testing
> ---
> 
> Additional unit tests.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

[jira] [Commented] (ATLAS-4408) Dynamic handling of failure in updating index

2021-10-10 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/ATLAS-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426950#comment-17426950
 ] 

ASF subversion and git services commented on ATLAS-4408:


Commit 261331bbd1fa8e50517421ab834f01f89f4997e1 in atlas's branch 
refs/heads/master from Radhika Kundam
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=261331b ]

ATLAS-4408: Dynamic handling of failure in updating index


> Dynamic handling of failure in updating index
> -
>
> Key: ATLAS-4408
> URL: https://issues.apache.org/jira/browse/ATLAS-4408
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Reporter: Radhika Kundam
>Assignee: Radhika Kundam
>Priority: Major
> Attachments: IndexRecovery.png, IndexRecovery_FunctionalFlow.png
>
>
> *Index failure resilience:* dynamic handling of failure in updating index 
> (i.e. HBase commit succeeds but index commit fails).
> In case of secondary persistence failure scenario, there will be 
> inconsistency with indexes for all the transactions failed at Solr. And to 
> repair that, the existing option is re-indexing all the data which is time 
> consuming as it involves indexing the entire database.
> To recover such inconsistencies we can use the *transaction write-ahead log 
> option*. By enabling write-ahead log(tx.log-tx), JanusGraph maintains all the 
> transaction log data which can be used to recover indices in case of 
> failures. With this approach, it’s extra overhead to maintain the log data 
> for all transactions but with this approach we can guarantee the system is 
> more resilient and proactive. So advantages of this approach can nullify the 
> overhead of maintaining log data.
> Design details as below.
>  # Start new service - IndexRecoveryService at Atlas startup.
>  ## Continuously monitor for Solr(Index Client) health for every retryTime 
> millisecs
>  ### If Solr is healthy and recovery start time is available, 
>   Start Transaction Recovery with available recovery start time(which is 
> noted when Solr became unhealthy)
>   Persist current recovery time as previous which can be used later by 
> passing as custom recovery time to start index recovery if required.
>   Reset current recovery start time
>   Continue with Solr health checkup.
>  ### If Solr is unhealthy and no recovery start time is available, 
>   Shutdown the existing transaction recovery process.
>   Note down the time which should be the next recovery start time and 
> persist in graph.
>   Continue with Solr health checkup.
> Configuration properties to be used for this feature.
> 1.To enable or disable index recovery(By default index recovery will be 
> enabled on Atlas startup)
>     *atlas.graph.enable.index.recovery=true*
>  2.To configure how frequently SOLR health check should be done
>     *atlas.graph.index.search.solr.status.retry.interval=*
>  3.To start index recovery by custom recovery time as user provided
>     *atlas.graph.index.search.solr.recovery.start.time=1630086622*
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4408) Dynamic handling of failure in updating index

2021-10-10 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/ATLAS-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426951#comment-17426951
 ] 

ASF subversion and git services commented on ATLAS-4408:


Commit 9021859cd90ec56e770c8e68907441807951e894 in atlas's branch 
refs/heads/branch-2.0 from Radhika Kundam
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=9021859 ]

ATLAS-4408: Dynamic handling of failure in updating index

(cherry picked from commit 261331bbd1fa8e50517421ab834f01f89f4997e1)


> Dynamic handling of failure in updating index
> -
>
> Key: ATLAS-4408
> URL: https://issues.apache.org/jira/browse/ATLAS-4408
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Reporter: Radhika Kundam
>Assignee: Radhika Kundam
>Priority: Major
> Attachments: IndexRecovery.png, IndexRecovery_FunctionalFlow.png
>
>
> *Index failure resilience:* dynamic handling of failure in updating index 
> (i.e. HBase commit succeeds but index commit fails).
> In case of secondary persistence failure scenario, there will be 
> inconsistency with indexes for all the transactions failed at Solr. And to 
> repair that, the existing option is re-indexing all the data which is time 
> consuming as it involves indexing the entire database.
> To recover such inconsistencies we can use the *transaction write-ahead log 
> option*. By enabling write-ahead log(tx.log-tx), JanusGraph maintains all the 
> transaction log data which can be used to recover indices in case of 
> failures. With this approach, it’s extra overhead to maintain the log data 
> for all transactions but with this approach we can guarantee the system is 
> more resilient and proactive. So advantages of this approach can nullify the 
> overhead of maintaining log data.
> Design details as below.
>  # Start new service - IndexRecoveryService at Atlas startup.
>  ## Continuously monitor for Solr(Index Client) health for every retryTime 
> millisecs
>  ### If Solr is healthy and recovery start time is available, 
>   Start Transaction Recovery with available recovery start time(which is 
> noted when Solr became unhealthy)
>   Persist current recovery time as previous which can be used later by 
> passing as custom recovery time to start index recovery if required.
>   Reset current recovery start time
>   Continue with Solr health checkup.
>  ### If Solr is unhealthy and no recovery start time is available, 
>   Shutdown the existing transaction recovery process.
>   Note down the time which should be the next recovery start time and 
> persist in graph.
>   Continue with Solr health checkup.
> Configuration properties to be used for this feature.
> 1.To enable or disable index recovery(By default index recovery will be 
> enabled on Atlas startup)
>     *atlas.graph.enable.index.recovery=true*
>  2.To configure how frequently SOLR health check should be done
>     *atlas.graph.index.search.solr.status.retry.interval=*
>  3.To start index recovery by custom recovery time as user provided
>     *atlas.graph.index.search.solr.recovery.start.time=1630086622*
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4408) Dynamic handling of failure in updating index

2021-10-10 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4408:
--
Fix Version/s: 2.3.0
   3.0.0

> Dynamic handling of failure in updating index
> -
>
> Key: ATLAS-4408
> URL: https://issues.apache.org/jira/browse/ATLAS-4408
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Radhika Kundam
>Assignee: Radhika Kundam
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: IndexRecovery.png, IndexRecovery_FunctionalFlow.png
>
>
> *Index failure resilience:* dynamic handling of failure in updating index 
> (i.e. HBase commit succeeds but index commit fails).
> In case of secondary persistence failure scenario, there will be 
> inconsistency with indexes for all the transactions failed at Solr. And to 
> repair that, the existing option is re-indexing all the data which is time 
> consuming as it involves indexing the entire database.
> To recover such inconsistencies we can use the *transaction write-ahead log 
> option*. By enabling write-ahead log(tx.log-tx), JanusGraph maintains all the 
> transaction log data which can be used to recover indices in case of 
> failures. With this approach, it’s extra overhead to maintain the log data 
> for all transactions but with this approach we can guarantee the system is 
> more resilient and proactive. So advantages of this approach can nullify the 
> overhead of maintaining log data.
> Design details as below.
>  # Start new service - IndexRecoveryService at Atlas startup.
>  ## Continuously monitor for Solr(Index Client) health for every retryTime 
> millisecs
>  ### If Solr is healthy and recovery start time is available, 
>   Start Transaction Recovery with available recovery start time(which is 
> noted when Solr became unhealthy)
>   Persist current recovery time as previous which can be used later by 
> passing as custom recovery time to start index recovery if required.
>   Reset current recovery start time
>   Continue with Solr health checkup.
>  ### If Solr is unhealthy and no recovery start time is available, 
>   Shutdown the existing transaction recovery process.
>   Note down the time which should be the next recovery start time and 
> persist in graph.
>   Continue with Solr health checkup.
> Configuration properties to be used for this feature.
> 1.To enable or disable index recovery(By default index recovery will be 
> enabled on Atlas startup)
>     *atlas.graph.enable.index.recovery=true*
>  2.To configure how frequently SOLR health check should be done
>     *atlas.graph.index.search.solr.status.retry.interval=*
>  3.To start index recovery by custom recovery time as user provided
>     *atlas.graph.index.search.solr.recovery.start.time=1630086622*
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4408) Dynamic handling of failure in updating index

2021-10-10 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4408:
--
Affects Version/s: 3.0.0
   2.2.0

> Dynamic handling of failure in updating index
> -
>
> Key: ATLAS-4408
> URL: https://issues.apache.org/jira/browse/ATLAS-4408
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Radhika Kundam
>Assignee: Radhika Kundam
>Priority: Major
> Attachments: IndexRecovery.png, IndexRecovery_FunctionalFlow.png
>
>
> *Index failure resilience:* dynamic handling of failure in updating index 
> (i.e. HBase commit succeeds but index commit fails).
> In case of secondary persistence failure scenario, there will be 
> inconsistency with indexes for all the transactions failed at Solr. And to 
> repair that, the existing option is re-indexing all the data which is time 
> consuming as it involves indexing the entire database.
> To recover such inconsistencies we can use the *transaction write-ahead log 
> option*. By enabling write-ahead log(tx.log-tx), JanusGraph maintains all the 
> transaction log data which can be used to recover indices in case of 
> failures. With this approach, it’s extra overhead to maintain the log data 
> for all transactions but with this approach we can guarantee the system is 
> more resilient and proactive. So advantages of this approach can nullify the 
> overhead of maintaining log data.
> Design details as below.
>  # Start new service - IndexRecoveryService at Atlas startup.
>  ## Continuously monitor for Solr(Index Client) health for every retryTime 
> millisecs
>  ### If Solr is healthy and recovery start time is available, 
>   Start Transaction Recovery with available recovery start time(which is 
> noted when Solr became unhealthy)
>   Persist current recovery time as previous which can be used later by 
> passing as custom recovery time to start index recovery if required.
>   Reset current recovery start time
>   Continue with Solr health checkup.
>  ### If Solr is unhealthy and no recovery start time is available, 
>   Shutdown the existing transaction recovery process.
>   Note down the time which should be the next recovery start time and 
> persist in graph.
>   Continue with Solr health checkup.
> Configuration properties to be used for this feature.
> 1.To enable or disable index recovery(By default index recovery will be 
> enabled on Atlas startup)
>     *atlas.graph.enable.index.recovery=true*
>  2.To configure how frequently SOLR health check should be done
>     *atlas.graph.index.search.solr.status.retry.interval=*
>  3.To start index recovery by custom recovery time as user provided
>     *atlas.graph.index.search.solr.recovery.start.time=1630086622*
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4408) Dynamic handling of failure in updating index

2021-10-10 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4408:
--
Labels: indexing  (was: )

> Dynamic handling of failure in updating index
> -
>
> Key: ATLAS-4408
> URL: https://issues.apache.org/jira/browse/ATLAS-4408
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Radhika Kundam
>Assignee: Radhika Kundam
>Priority: Major
>  Labels: indexing
> Fix For: 3.0.0, 2.3.0
>
> Attachments: IndexRecovery.png, IndexRecovery_FunctionalFlow.png
>
>
> *Index failure resilience:* dynamic handling of failure in updating index 
> (i.e. HBase commit succeeds but index commit fails).
> In case of secondary persistence failure scenario, there will be 
> inconsistency with indexes for all the transactions failed at Solr. And to 
> repair that, the existing option is re-indexing all the data which is time 
> consuming as it involves indexing the entire database.
> To recover such inconsistencies we can use the *transaction write-ahead log 
> option*. By enabling write-ahead log(tx.log-tx), JanusGraph maintains all the 
> transaction log data which can be used to recover indices in case of 
> failures. With this approach, it’s extra overhead to maintain the log data 
> for all transactions but with this approach we can guarantee the system is 
> more resilient and proactive. So advantages of this approach can nullify the 
> overhead of maintaining log data.
> Design details as below.
>  # Start new service - IndexRecoveryService at Atlas startup.
>  ## Continuously monitor for Solr(Index Client) health for every retryTime 
> millisecs
>  ### If Solr is healthy and recovery start time is available, 
>   Start Transaction Recovery with available recovery start time(which is 
> noted when Solr became unhealthy)
>   Persist current recovery time as previous which can be used later by 
> passing as custom recovery time to start index recovery if required.
>   Reset current recovery start time
>   Continue with Solr health checkup.
>  ### If Solr is unhealthy and no recovery start time is available, 
>   Shutdown the existing transaction recovery process.
>   Note down the time which should be the next recovery start time and 
> persist in graph.
>   Continue with Solr health checkup.
> Configuration properties to be used for this feature.
> 1.To enable or disable index recovery(By default index recovery will be 
> enabled on Atlas startup)
>     *atlas.graph.enable.index.recovery=true*
>  2.To configure how frequently SOLR health check should be done
>     *atlas.graph.index.search.solr.status.retry.interval=*
>  3.To start index recovery by custom recovery time as user provided
>     *atlas.graph.index.search.solr.recovery.start.time=1630086622*
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73641: Repair Composite indexes

2021-10-10 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73641/#review223595
---


Fix it, then Ship it!





graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Lines 38 (patched)


nit: unused import; consider removing



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Lines 324 (patched)


nit: add import statement for SchemaStatus and SchemaAction.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Lines 329 (patched)


Add more detail to log statement:

LOG.info("Index schema status updated for index: {}, from {} to {}.", 
index.name(), SchemaStatus.REGISTERED, SchemaStatus.ENABLED);



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Lines 332 (patched)


consider adding else if block to handle WARN logging for "INSTALLED" schema 
status



repository/src/main/java/org/apache/atlas/repository/patches/RepairCompositeIndexPatch.java
Lines 27 (patched)


consider renaming class to "UpdateCompositeIndexStatusPatch", since this 
JAVA patch deals with UPDATING the schema status of certain composite indexes.


- Sarath Subramanian


On Oct. 8, 2021, 9:05 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73641/
> ---
> 
> (Updated Oct. 8, 2021, 9:05 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4447
> https://issues.apache.org/jira/browse/ATLAS-4447
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> If by any chance, commposite index are not enabled, this patch fixes the 
> composite index from REGISTERED state to ENABLED state
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  7e3b2f440 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  1cc7f8b55 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
>  d30971afc 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/RepairCompositeIndexPatch.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/73641/diff/1/
> 
> 
> Testing
> ---
> 
> manually tested
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>