----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73430/#review223178 -----------------------------------------------------------
intg/src/main/java/org/apache/atlas/ApplicationProperties.java Lines 349 (patched) <https://reviews.apache.org/r/73430/#comment312271> What is the need for this if condition: boolean solrWaitSearcherValue = getBoolean(SOLR_WAIT_SEARCHER_CONF, DEFAULT_SOLR_WAIT_SEARCHER); repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java Line 880 (original), 880 (patched) <https://reviews.apache.org/r/73430/#comment312272> Not necessary. repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java Lines 76 (patched) <https://reviews.apache.org/r/73430/#comment312273> Sorround with: try { LOG.debug("Waiting for indexes to catch-up: Starting..."); Thread.sleep(10000); } catch (InterruptedException e) { e.printStackTrace(); } finally { LOG.debug("Waiting for indexes to catch-up: Done!"); } repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java Line 146 (original), 146 (patched) <https://reviews.apache.org/r/73430/#comment312274> Not necessary. repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java Line 57 (original), 57 (patched) <https://reviews.apache.org/r/73430/#comment312275> Not necessary. webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java Line 80 (original), 80 (patched) <https://reviews.apache.org/r/73430/#comment312276> Not necessary. webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java Line 143 (original), 145 (patched) <https://reviews.apache.org/r/73430/#comment312277> Not necessary. webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java Lines 208 (patched) <https://reviews.apache.org/r/73430/#comment312278> Sorround with: try { LOG.debug("Waiting for indexes to catch-up: Starting..."); Thread.sleep(10000); } catch (InterruptedException e) { e.printStackTrace(); } finally { LOG.debug("Waiting for indexes to catch-up: Done!"); } webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java Line 221 (original), 224 (patched) <https://reviews.apache.org/r/73430/#comment312279> Not necessary. - Ashutosh Mestry On June 22, 2021, 4:29 p.m., Sarath Subramanian wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73430/ > ----------------------------------------------------------- > > (Updated June 22, 2021, 4:29 p.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nikhil Bonte, and Pinal Shah. > > > Bugs: ATLAS-4340 > https://issues.apache.org/jira/browse/ATLAS-4340 > > > Repository: atlas > > > Description > ------- > > In Atlas when a transaction is committed, the entries are committed to HBase > (primary storage) and Solr (indexing storage). A transaction is rolled-back > if the primary storage commit fails, on the other hand when the secondary > commit fails (solr), the transaction is not-rolled back and logged as warning > and it is recommended to use reindex to repair the missing index documents. > This behavior is due to the fact that the primary storage is the source of > truth and indexes can be rebuild. > > In Janusgraph, there is a property for Solr to make solr commits async. This > is set to true in Atlas making every commit to wait until the solr commit is > successful. This will have a negative impact on performance and is > recommended to be false by default. > > Property: index.[X].solr.wait-searcher > > When mutating - wait for the index to reflect new mutations before returning. > This can have a negative impact on performance. > > > This Jira is about setting the default value for above property to FALSE and > can be overridden if need arises. > > > Diffs > ----- > > intg/src/main/java/org/apache/atlas/ApplicationProperties.java bf97ab146 > repository/src/test/java/org/apache/atlas/BasicTestSetup.java a1d7b62a5 > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > a9fbd435a > > repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java > 121dca9f5 > > repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java > 764568915 > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java > a3be5f4ed > webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java > 962345b22 > > > Diff: https://reviews.apache.org/r/73430/diff/1/ > > > Testing > ------- > > 1. Precommit Test: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/688/ > 2. Build Atlas with embedded Hbase/Solr profile and validated basic sanity > tests - running quick start, basic search, tag propagation > 3. Performance Test details: > > > Run with default settings - Solr wait-searcher property enabled - true > (without patch) > ---------------------------------------------------------------------- > Start Time : Tue Jun 15 22:26:58 PDT 2021 > End Time : Fri Jun 18 02:32:34 PDT 2021 > Messages Processed : 91,225 > Time Taken : 52 hours 5 mins > Rate : ~ 29.2 messages/minute > > > Run with disabled Solr wait-searcher property (will improve solr commit time > making it async) - with patch > ----------------------------------------------------------------------------------------------------------- > Start Time : Mon Jun 14 13:30:04 PDT 2021 > End Time : Tue Jun 15 17:23:56 PDT 2021 > Messages Processed : 91,225 > Time Taken : 27 hours 54 mins > Rate : ~ 54.5 messages/minute > > > We see almost 50% perf imrpovement with this change. > > > Thanks, > > Sarath Subramanian > >
