This is an automated email from the ASF dual-hosted git repository.
daim pushed a change to branch OAK-10199
in repository https://gitbox.apache.org/repos/asf/jackrabbit-oak.git
omit ca1c451a84 OAK-10199 : added logic to skip non garbage documents
omit e2951693d1 OAK-10199 : added unit cases to handle concurrent prop
update and escaped properties update
omit 551d21b24f OAK-10199 : handled escaped properties while deleting them
omit ac6e292b1b OAK-10199 : added test case to continue with GC even if
there is crash
omit 4ebeff4cc7 OAK-10199 : fixed logic to include previously garbage
collected documents if updated recently
omit f774427333 OAK-10199 : fixed the VG INIT test
omit 3e40e5e35c OAK-10199 : fixed the VGC IT test cases failures for RDB
omit 29a17dba0e OAK-10199 : fixed the test case with different revision for
DetailedGC
omit 6748df6823 OAK-10199 : added check to include oldestId when running
detailedGc very first time
omit f274163408 OAK-10199 : added test cases to fetch 5000+ modified docs
in loop and verify them
omit 70413d572c OAK-10199 : minor refactoring
omit 1f533a8c72 OAK-10199 : updated logic to fetch nodes by sorting them on
the basis of _modified & _id
omit 754c1cb90c OAK-10199 : fixed code smells as suggested by Sonar
omit 53b0742896 OAK-10199 : ignore documents which doesn't have _modified
field in mongo while fetching modifiedDocs
omit 3b258bc942 OAK-10199 : used bulk findAndModify api to perform garbage
cleanup
omit d70566b96e OAK-10199 : override getModifiedDocs() for RDB and added
unit cases for deletedProps
omit cfecc9a52c OAK-10199 : provided support for feature toggle & osgi
config for detailed gc
omit d154f0ba69 OAK-10199 : disable the detailGc in tearDown to avoid
side-effects
omit dc9f483623 OAK-10199 : initial sketch of detail gc skeleton
add 5aa3c1b4dd oak-search-elastic: field values should be deduplicated
(#1006)
add e50de3a317 OAK-10226: fix lucene->elastic conversion for WordDelimiter
(#1003)
add e50f432887 OAK-10313: Identify revisions created by late-write scenario
add 41e78a3a77 OAK-10313: Identify revisions created by late-write scenario
add e2bbbebf49 Merge branch 'trunk' into OAK-10313
add c5fe2b73cd OAK-10313: Identify revisions created by late-write scenario
add 3669b31f39 Merge branch 'trunk' into OAK-10313
add 56366607a4 Merge pull request #995 from mreutegg/OAK-10313
add cbfacdb90f Initial commit of pipelined download strategy.
add c505124a35 Minor fixes
add 03e3be8992 WIP
add 58a36a1bd2 Merge remote-tracking branch 'upstream/trunk' into
GRANITE-45911
add f805fdab28 Add recovery from broken MongoDB connections to downloader.
add 9bfb9fc54a Merge remote-tracking branch 'upstream/trunk' into
GRANITE-45911
add 45d0eaf291 Shutdown gracefully if one of the dump stages fails with an
exception.
add 9d1382ccb0 Add documentation. Change the configuration for the retrial
mechanism for MongoDB connection failures: instead of number of retrials, use
the amount of time to keep trying before giving up.
add db824bd6dc Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add 832074fe64 Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add 7384304e73 Add support for auto-tuning working set memory based on
total available memory to the indexer. Switch to using system properties to
configure the pipelined strategy (instead of env variables).
add 981e673e6c Always use read preference secondaryPreferred for
downloading from Mongo.
add 0e10939319 Address reviews comments.
add 8633be3e60 Use 0 and Long.MAX_VALUE as boundaries for download range,
no need to query Mongo to determine the earliest and latest values of _modified.
add 5edda975aa Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add c0b80afe51 Add a new stage to merge sorted files. Log how much time
the download and transform threads spend waiting to enqueue their outputs in
the out queues (this indicates that the stages after them are too slow).
Collect and log metrics describing the download and transform stages, like
mongo documents downloaded, node state entries extracted, filtered, mongo
documents that do not match a node state (garbage) and a few others. Use only
FileUtils.byteCountToDisplaySize() to p [...]
add ea7ce8a70f Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add 2b84a30d44 Improve collection and logging of metrics.
add c68fad0dcc Add unit test for merge-sort task.
add 93f9eacd85 Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add 8c5dc547e8 Add more tests, including a first draft of an integration
test. Refactor code.
add 787a4d575c Fix
add 62697ca44e Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add 66d176bb90 Add an integration test
add 6387d4dc5b Add license header.
add a9dd47546f Add license header.
add ab27625a83 Rename test files.
add fabea754fd Do not filter on _modified != null when doing
non-recoverable download.
add 1cf44cf70c Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add a7f48a0370 Fixes based on review comments.
add 5590997ded Bound the size of the histograms.
add ab06697edb Add missing license files.
add 2839cc9d45 Improve logging and code clean of bounded histogram.
add ef2182e4aa Display human readable byte counts with 2 decimal places
instead of abbreviating to closest integer value.
add 27f9b710ef Improved logging.
add 4f92e897d9 Refactoring
add 4c33e932b4 Remove changes unrelated to PR.
add ec2b12fd8e Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add a4b7bf1016 Add tests for error handling when configuration properties
have invalid values. Minor refactoring.
add 11f308c86f Add test for when the path predicate does not match any
existing path in the document store. Address other review comments.
add 81eaeead4a Update
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMergeSortTask.java
add 560dd56d5d Update
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMongoDownloadTask.java
add 00923a38c0 Merge remote-tracking branch 'origin/OAK-10294' into
OAK-10294
add 074e4af804 Add unit test to test recovery from broken mongo
connections. Fix: when a connection to Mongo is lost, the documents that were
collected in a block but not yet enqueued were being lost.
add 70e6b89a86 Add missing license.
add 25c01b8176 Merge pull request #979 from nfsantos/OAK-10294
add 8677db8d98 OAK-10331 add caller information to different warnings
(#1010)
add 965d6703b5 OAK-10337: mvn jetty:run fails to start oak-web (#1016)
add db3bf04066 OAK-10307: re-add guava to several subprojects to re-enable
isolated compilation (will be removed later on) - fix import range
add e1d3e3e8db OAK-10338 - Do not use platform-dependent newlines in for
files generated when creating a FlatFileStore (#1018)
add 6b1686326e OAK-10307: re-add guava to several subprojects to re-enable
isolated compilation (will be removed later on) - fix import range
add 288e81cacb OAK-10342: Update Apache Parent POM to version 30
add 1a6acc0a8b OAK-10338: Change test to tolerate different line endings
add 9ea124cfbb OAK-10327 Embedded dependencies should have "provided"
scope (#1004)
add 53bc40653e OAK-10326: remove baseline comparison version (back to
default)
add 4d7cf4b58d Revert "OAK-10342: Update Apache Parent POM to version 30"
add 03b703e91f OAK-10332 Allow reusing FilterProviderImpl outside OSGi
context (#1014)
add 6005e9ae70 OAK-10199 : initial sketch of detail gc skeleton
add 3a3cd82f1c OAK-10199 : disable the detailGc in tearDown to avoid
side-effects
add 6330016e33 OAK-10199 : provided support for feature toggle & osgi
config for detailed gc
add 29971b2406 OAK-10199 : override getModifiedDocs() for RDB and added
unit cases for deletedProps
add dfbfac5fa3 OAK-10199 : used bulk findAndModify api to perform garbage
cleanup
add d6a92ddc2a OAK-10199 : ignore documents which doesn't have _modified
field in mongo while fetching modifiedDocs
add 04b00713c9 OAK-10199 : fixed code smells as suggested by Sonar
add 84d5c0cca5 OAK-10199 : updated logic to fetch nodes by sorting them on
the basis of _modified & _id
add 6707b71421 OAK-10199 : minor refactoring
add 6b623ea5bc OAK-10199 : added test cases to fetch 5000+ modified docs
in loop and verify them
add fbabf40927 OAK-10199 : added check to include oldestId when running
detailedGc very first time
add b7551f4b4a OAK-10199 : fixed the test case with different revision for
DetailedGC
add 416e913e14 OAK-10199 : fixed the VGC IT test cases failures for RDB
add d1fb4cbb49 OAK-10199 : fixed the VG INIT test
add d63e635ae8 OAK-10199 : fixed logic to include previously garbage
collected documents if updated recently
add 54d5e8b30a OAK-10199 : added test case to continue with GC even if
there is crash
add c812ad818d OAK-10199 : handled escaped properties while deleting them
add c984745e53 OAK-10199 : added unit cases to handle concurrent prop
update and escaped properties update
add 0506c012a6 OAK-10199 : added logic to skip non garbage documents
add e8aded27e8 OAK-10199 : fixed query to avoid skipping documents with
greater _modified timestamp
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (ca1c451a84)
\
N -- N -- N refs/heads/OAK-10199 (e8aded27e8)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../principalbased/impl/FilterProviderImpl.java | 23 +-
.../impl/AbstractPrincipalBasedTest.java | 4 +-
oak-blob-cloud-azure/pom.xml | 4 +
.../org/apache/jackrabbit/oak/commons/IOUtils.java | 21 +
.../jackrabbit/oak/commons/sort/ExternalSort.java | 4 +-
.../org/apache/jackrabbit/oak/query/QueryImpl.java | 7 +-
oak-examples/webapp/pom.xml | 19 +-
oak-parent/pom.xml | 2 -
oak-run-commons/pom.xml | 7 +
.../indexer/document/DocumentStoreIndexerBase.java | 63 +-
.../indexer/document/NodeStateEntryTraverser.java | 7 +-
.../document/NodeStateEntryTraverserFactory.java | 5 +-
.../flatfile/FlatFileNodeStoreBuilder.java | 39 +-
.../document/flatfile/FlatFileStoreUtils.java | 8 +-
.../MultithreadedTraverseWithSortStrategy.java | 10 +-
.../document/flatfile/NodeStateEntryWriter.java | 26 +-
.../document/flatfile/StoreAndSortStrategy.java | 11 +-
.../document/flatfile/TraverseAndSortTask.java | 10 +-
.../flatfile/TraverseWithSortStrategy.java | 19 +-
.../flatfile/pipelined/BoundedHistogram.java | 89 +++
.../document/flatfile/pipelined/ConfigHelper.java} | 44 +-
.../document/flatfile/pipelined/DownloadRange.java | 74 +++
.../flatfile/pipelined/NodeStateEntryBatch.java | 96 +++
.../NodeStateHolder.java} | 39 +-
.../{ => pipelined}/PathElementComparator.java | 32 +-
.../flatfile/pipelined/PipelinedMergeSortTask.java | 140 ++++
.../pipelined/PipelinedMongoDownloadTask.java | 327 ++++++++++
.../flatfile/pipelined/PipelinedSortBatchTask.java | 155 +++++
.../flatfile/pipelined/PipelinedStrategy.java | 433 +++++++++++++
.../flatfile/pipelined/PipelinedTransformTask.java | 246 +++++++
.../document/flatfile/pipelined/SortKey.java | 89 +++
.../pipelined/TransformStageStatistics.java | 171 +++++
.../document/mongo/MongoDocumentStoreHelper.java | 9 +
.../document/mongo/MongoDocumentTraverser.java | 65 +-
.../plugins/document/mongo/TraversingRange.java | 81 +++
.../document/flatfile/FlatFileStoreTest.java | 8 +-
.../MultithreadedTraverseWithSortStrategyTest.java | 4 +-
.../document/flatfile/TraverseAndSortTaskTest.java | 2 +-
.../flatfile/pipelined/BoundedHistogramTest.java | 56 ++
.../pipelined/NodeStateEntryBatchTest.java | 88 +++
.../document/flatfile/pipelined/PipelinedIT.java | 214 +++++++
.../pipelined/PipelinedMergeSortTaskTest.java | 121 ++++
.../pipelined/PipelinedMongoDownloadTaskTest.java | 104 +++
.../pipelined/PipelinedSortBatchTaskTest.java | 186 ++++++
.../test/resources/pipelined/merge-expected.json | 6 +
.../test/resources/pipelined/merge-stage-1.json | 3 +
.../test/resources/pipelined/merge-stage-2.json | 3 +
...rphanedNodeCheck.java => ConsistencyCheck.java} | 80 +--
.../plugins/document/check/DocumentStoreCheck.java | 18 +-
.../oak/run/DocumentStoreCheckCommand.java | 9 +
.../document/mongo/DocumentTraverserTest.java | 2 +-
.../index/ElasticCustomAnalyzerMappings.java | 43 ++
.../index/elastic/index/ElasticDocument.java | 10 +-
.../index/elastic/ElasticAbstractQueryTest.java | 15 +
.../plugins/index/elastic/ElasticContentTest.java | 24 +
oak-search-mt/pom.xml | 1 +
.../plugins/index/FullTextAnalyzerCommonTest.java | 37 ++
oak-segment-azure/pom.xml | 1 +
oak-segment-tar/pom.xml | 14 +-
oak-store-document/pom.xml | 1 +
.../oak/plugins/document/Consistency.java | 138 ++++
.../plugins/document/VersionGCRecommendations.java | 85 ++-
.../oak/plugins/document/VersionGCSupport.java | 39 +-
.../plugins/document/VersionGarbageCollector.java | 65 +-
.../document/mongo/MongoVersionGCSupport.java | 38 +-
.../plugins/document/rdb/RDBVersionGCSupport.java | 57 +-
.../oak/plugins/document/FailingDocumentStore.java | 37 +-
.../oak/plugins/document/VersionGCInitTest.java | 13 +-
.../oak/plugins/document/VersionGCSupportTest.java | 8 +-
.../oak/plugins/document/VersionGCTest.java | 8 +-
.../document/VersionGarbageCollectorIT.java | 96 ++-
.../plugins/document/WriteAfterRecoveryTest.java | 707 +++++++++++++++++++++
72 files changed, 4300 insertions(+), 420 deletions(-)
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/BoundedHistogram.java
copy
oak-run-commons/src/{test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/CountingIterable.java
=>
main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/ConfigHelper.java}
(53%)
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/DownloadRange.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/NodeStateEntryBatch.java
copy
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/{SimpleNodeStateHolder.java
=> pipelined/NodeStateHolder.java} (55%)
copy
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/{
=> pipelined}/PathElementComparator.java (73%)
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMergeSortTask.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMongoDownloadTask.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedSortBatchTask.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedStrategy.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedTransformTask.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/SortKey.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/TransformStageStatistics.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/plugins/document/mongo/TraversingRange.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/BoundedHistogramTest.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/NodeStateEntryBatchTest.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedIT.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMergeSortTaskTest.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMongoDownloadTaskTest.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedSortBatchTaskTest.java
create mode 100644
oak-run-commons/src/test/resources/pipelined/merge-expected.json
create mode 100644
oak-run-commons/src/test/resources/pipelined/merge-stage-1.json
create mode 100644
oak-run-commons/src/test/resources/pipelined/merge-stage-2.json
copy
oak-run/src/main/java/org/apache/jackrabbit/oak/plugins/document/check/{OrphanedNodeCheck.java
=> ConsistencyCheck.java} (53%)
create mode 100644
oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/Consistency.java
create mode 100644
oak-store-document/src/test/java/org/apache/jackrabbit/oak/plugins/document/WriteAfterRecoveryTest.java
