Hey Michael, Good to hear because we are updating to 1.6.6. in 3 weeks! Let's hope it helps.
About the .bak files, do I understand it correctly then that the .bak files are
only created when something went wrong on shutdown?
Greets,
Roy
> On 5 Mar 2018, at 16:59, Michael Dürig <mic...@gmail.com> wrote:
>
> Roy,
>
> One thing that might help is upgrading to 1.6.6 to get the fix for
> OAK-6642 /OAK-6110. With that fix you should be able to run within a
> 2GB heap and compaction times should go down.
>
> Re. the bak files: from an operational POV it is safe to remove them.
> Those files are kept in the case an instances in shut down in an non
> clean way and needs to recover the most recent tar file that hasn't
> been properly closed. There is up to 5 seconds worth of data that
> might get lost in that process. The bak files *might* contain parts of
> this data and it *might* be possible to recover *some* of it manually.
> If this is not needed (I haven't seen a case where it was so far), you
> can safely remove them.
>
> Michael
>
> .
>
> On 5 March 2018 at 14:44, Roy Teeuwen <r...@teeuwen.be> wrote:
>> Hey guys,
>>
>> We have upgraded to Oak 1.6.5 (AEM 6.3) and we have noticed that the offline
>> compaction is a lot slower than it was initially in AEM 6.1. While we had
>> compaction of 5-15 mins for 15GB at first, it now takes half an hour. Is
>> there any performance tweaking / parameters we can use to speed it up? Below
>> you can find the script we use to do daily offline compaction and a log file
>> of one of those times. Usually it takes away one GB per day. Seeing as we
>> have the possibility to take down the system once a night, we don't have to
>> solely rely on online compaction.
>>
>> Another question I have with this, would it be ok that we remove the .bak
>> files automatically after the compaction has ran? Or why do they stay in the
>> folder?
>>
>> Script:
>> ```
>> #!/bin/bash
>>
>> # AEM config
>> AEM_PATH="/opt/aem/author"
>> AEM_USER="aem-user"
>> CRX_QUICKSTART_PATH="${AEM_PATH}/crx-quickstart"
>> REPOSITORY_PATH="${CRX_QUICKSTART_PATH}/repository"
>> SEGMENTSTORE_PATH="${REPOSITORY_PATH}/segmentstore"
>> AEM_STATUS="${CRX_QUICKSTART_PATH}/bin/status"
>> AEM_START="${CRX_QUICKSTART_PATH}/bin/start"
>> AEM_STOP="${CRX_QUICKSTART_PATH}/bin/stop"
>>
>> # OAK Run config
>> OAK_RUN_JAR="/opt/aem_dependencies/aem/oak-run-1.6.5.jar"
>> HEAPSIZE="-Xmx12228M"
>> UPDATE_LIMIT="-Dupdate.limit=10000"
>> COMPRESS_INTERVAL="-Dcompress-interval=100000"
>> COMPACTION_PROGRESS_LOG="-Dcompaction-progress-log=150000"
>> MEMORY_MAPPED="-Dtar.memoryMapped=true"
>> OAK_RUN_PARAMETERS="${HEAPSIZE} ${UPDATE_LIMIT} ${COMPRESS_INTERVAL}
>> ${COMPACTION_PROGRESS_LOG} ${MEMORY_MAPPED}"
>> OAK_RUN="java ${OAK_RUN_PARAMETERS} -jar ${OAK_RUN_JAR}"
>>
>> if ! [ "$(whoami)" = "${AEM_USER}" ]
>> then
>> echo "Error: Maintenance task needs to be run as AEM user (${AEM_USER})."
>> exit 1
>> fi
>>
>> echo "==> Starting maintenance ($(date))"
>>
>> echo "==> Make sure AEM is not running"
>> if ${AEM_STATUS}
>> then
>> echo "==> Stopping AEM.."
>> ${AEM_STOP}
>> fi
>>
>> echo "==> Cleanup unrefrenced checkpoints"
>> ${OAK_RUN} checkpoints ${SEGMENTSTORE_PATH} || exit 1
>> ${OAK_RUN} checkpoints ${SEGMENTSTORE_PATH} rm-unreferenced || exit 1
>>
>> echo "==> Compact TAR segmentstore"
>> ${OAK_RUN} compact ${SEGMENTSTORE_PATH} || exit 1
>>
>> echo "==> Starting AEM"
>> ${AEM_START}
>>
>> echo "==> Maintenance completed ($(date))"
>>
>> exit 0
>>
>> ```
>>
>> Logs:
>>
>> ```
>> ==> Stopping AEM..
>> 05.03.2018 04:00:01.716 *INFO * [main] Setting sling.home=. (command line)
>> 05.03.2018 04:00:01.743 *INFO * [main] Sent 'stop' to /127.0.0.1:45726: OK
>> Application not running
>> ==> Cleanup unrefrenced checkpoints
>> Apache Jackrabbit Oak 1.6.5
>> Checkpoints /opt/aem/author/crx-quickstart/repository/segmentstore
>> - 3f22ad0f-3b78-4764-a697-c323f6d95180 created 2018-03-05 04:26:30.67
>> expires 2020-11-29 04:26:30.67
>> - 5caca09a-9d6b-426d-8166-e967ca1fbf7e created 2018-03-05 04:26:30.567
>> expires 2020-11-29 04:26:30.567
>> Found 2 checkpoints
>> Apache Jackrabbit Oak 1.6.5
>> Checkpoints /opt/aem/author/crx-quickstart/repository/segmentstore
>> Referenced checkpoint from /:async@async is
>> 3f22ad0f-3b78-4764-a697-c323f6d95180
>> Referenced checkpoint from /:async@fulltext-async is
>> 5caca09a-9d6b-426d-8166-e967ca1fbf7e
>> Removed 0 checkpoints in 44ms.
>> ==> Compact TAR segmentstore
>> Apache Jackrabbit Oak 1.6.5
>> Compacting /opt/aem/author/crx-quickstart/repository/segmentstore
>> With default access mode
>> before
>> Sun Mar 04 04:48:27 CET 2018, data05195a.tar
>> Sun Mar 04 04:51:46 CET 2018, data05213a.tar
>> Sun Mar 04 04:37:16 CET 2018, data05173a.tar
>> Sun Mar 04 04:51:22 CET 2018, data05211a.tar
>> Sun Mar 04 04:43:13 CET 2018, data05182a.tar
>> Sun Mar 04 04:55:57 CET 2018, data05226a.tar
>> Mon Mar 05 03:00:57 CET 2018, data05229a.tar
>> Sun Mar 04 04:36:03 CET 2018, data05171a.tar
>> Sun Mar 04 04:52:42 CET 2018, data05218a.tar
>> Sun Mar 04 04:34:42 CET 2018, data05169a.tar
>> Tue Jan 16 06:25:35 CET 2018, data01538a.tar.bak
>> Sun Mar 04 04:34:07 CET 2018, data05168a.tar
>> Sun Mar 04 04:49:31 CET 2018, data05201a.tar
>> Sun Mar 04 04:53:31 CET 2018, data05222a.tar
>> Sun Mar 04 06:11:04 CET 2018, data05228a.tar
>> Sun Mar 04 04:46:31 CET 2018, data05188a.tar
>> Sun Mar 04 04:50:08 CET 2018, data05204a.tar
>> Sun Mar 04 04:48:36 CET 2018, data05196a.tar
>> Thu Dec 14 07:14:54 CET 2017, repo.lock
>> Sun Mar 04 04:43:54 CET 2018, data05183a.tar
>> Sun Mar 04 04:49:08 CET 2018, data05199a.tar
>> Sun Mar 04 04:49:43 CET 2018, data05202a.tar
>> Fri Mar 02 04:57:25 CET 2018, data05022a.tar.2.bak
>> Sun Mar 04 04:41:19 CET 2018, data05179a.tar
>> Mon Mar 05 04:26:34 CET 2018, journal.log
>> Fri Mar 02 04:57:24 CET 2018, data05022a.tar.bak
>> Sun Mar 04 04:53:55 CET 2018, data05224a.tar
>> Sun Mar 04 04:49:55 CET 2018, data05203a.tar
>> Sun Mar 04 04:53:03 CET 2018, data05220a.tar
>> Mon Mar 05 04:26:45 CET 2018, manifest
>> Sun Mar 04 04:49:20 CET 2018, data05200a.tar
>> Sun Mar 04 04:42:30 CET 2018, data05181a.tar
>> Sun Mar 04 04:45:37 CET 2018, data05186a.tar
>> Sun Mar 04 04:55:57 CET 2018, data05227a.tar
>> Sun Mar 04 04:48:59 CET 2018, data05198a.tar
>> Sun Mar 04 04:37:39 CET 2018, data05174a.tar
>> Sun Mar 04 04:52:19 CET 2018, data05216a.tar
>> Sun Mar 04 04:47:43 CET 2018, data05191a.tar
>> Sun Mar 04 04:48:47 CET 2018, data05197a.tar
>> Sun Mar 04 04:45:01 CET 2018, data05185a.tar
>> Sun Mar 04 04:38:20 CET 2018, data05175a.tar
>> Sun Mar 04 04:50:42 CET 2018, data05207a.tar
>> Sun Mar 04 04:40:31 CET 2018, data05178a.tar
>> Sun Mar 04 04:52:30 CET 2018, data05217a.tar
>> Sun Mar 04 04:46:56 CET 2018, data05189a.tar
>> Sun Mar 04 04:39:05 CET 2018, data05176a.tar
>> Sun Mar 04 04:35:12 CET 2018, data05170a.tar
>> Sun Mar 04 04:50:33 CET 2018, data05206a.tar
>> Sun Mar 04 04:47:22 CET 2018, data05190a.tar
>> Sun Mar 04 04:51:34 CET 2018, data05212a.tar
>> Sun Mar 04 04:54:25 CET 2018, data05225a.tar
>> Sun Mar 04 04:36:55 CET 2018, data05172a.tar
>> Sun Mar 04 04:50:20 CET 2018, data05205a.tar
>> Sun Mar 04 04:51:58 CET 2018, data05214a.tar
>> Mon Mar 05 04:26:41 CET 2018, data05231a.tar
>> Mon Mar 05 04:19:14 CET 2018, data05230a.tar
>> Sun Mar 04 04:53:43 CET 2018, data05223a.tar
>> Sun Mar 04 04:44:31 CET 2018, data05184a.tar
>> Sun Mar 04 04:47:53 CET 2018, data05192a.tar
>> Sun Mar 04 04:39:47 CET 2018, data05177a.tar
>> Sun Mar 04 04:48:17 CET 2018, data05194a.tar
>> Sun Mar 04 04:52:09 CET 2018, data05215a.tar
>> Sun Mar 04 04:48:05 CET 2018, data05193a.tar
>> Sun Mar 04 04:52:52 CET 2018, data05219a.tar
>> Sun Mar 04 04:53:20 CET 2018, data05221a.tar
>> Sun Mar 04 04:50:52 CET 2018, data05208a.tar
>> Sun Mar 04 04:46:07 CET 2018, data05187a.tar
>> Sun Mar 04 04:51:12 CET 2018, data05210a.tar
>> Sun Mar 04 04:56:04 CET 2018, gc.log
>> Sun Mar 04 04:33:34 CET 2018, data05167a.tar
>> Sun Mar 04 04:41:47 CET 2018, data05180a.tar
>> Sun Mar 04 04:51:02 CET 2018, data05209a.tar
>> size 17.1 GB (17094875037 bytes)
>> -> compacting
>> 04:26:46.520 [main] WARN o.a.jackrabbit.oak.segment.Compactor - Deprecated
>> argument 'compress-interval', please use 'oak.segment.compaction.cacheSize'
>> instead.
>> -> cleaning up
>> -> writing new journal.log: bf675b66-e8e6-4bf4-a982-dc7316323e9c:859 root
>> 1520222043417
>>
>> after
>> Mon Mar 05 04:51:23 CET 2018, data05285a.tar
>> Mon Mar 05 04:36:53 CET 2018, data05244a.tar
>> Mon Mar 05 04:48:54 CET 2018, data05276a.tar
>> Mon Mar 05 04:43:10 CET 2018, data05254a.tar
>> Mon Mar 05 04:53:55 CET 2018, data05291a.tar
>> Mon Mar 05 04:48:31 CET 2018, data05274a.tar
>> Mon Mar 05 04:50:47 CET 2018, data05283a.tar
>> Mon Mar 05 04:40:27 CET 2018, data05249a.tar
>> Mon Mar 05 04:52:22 CET 2018, data05288a.tar
>> Mon Mar 05 04:49:30 CET 2018, data05278a.tar
>> Mon Mar 05 04:28:56 CET 2018, data05233a.tar
>> Mon Mar 05 04:31:18 CET 2018, data05236a.tar
>> Mon Mar 05 04:52:02 CET 2018, data05287a.tar
>> Tue Jan 16 06:25:35 CET 2018, data01538a.tar.bak
>> Mon Mar 05 04:47:51 CET 2018, data05272a.tar
>> Mon Mar 05 04:36:02 CET 2018, data05243a.tar
>> Mon Mar 05 04:35:14 CET 2018, data05242a.tar
>> Mon Mar 05 04:42:46 CET 2018, data05253a.tar
>> Mon Mar 05 04:44:55 CET 2018, data05260a.tar
>> Mon Mar 05 04:39:07 CET 2018, data05247a.tar
>> Mon Mar 05 04:46:50 CET 2018, data05268a.tar
>> Mon Mar 05 04:44:06 CET 2018, data05256a.tar
>> Mon Mar 05 04:45:33 CET 2018, data05263a.tar
>> Mon Mar 05 04:53:22 CET 2018, data05290a.tar
>> Thu Dec 14 07:14:54 CET 2017, repo.lock
>> Mon Mar 05 04:50:19 CET 2018, data05281a.tar
>> Mon Mar 05 04:49:13 CET 2018, data05277a.tar
>> Mon Mar 05 04:45:51 CET 2018, data05264a.tar
>> Mon Mar 05 04:33:40 CET 2018, data05240a.tar
>> Mon Mar 05 04:46:08 CET 2018, data05265a.tar
>> Mon Mar 05 04:32:55 CET 2018, data05239a.tar
>> Mon Mar 05 04:41:04 CET 2018, data05250a.tar
>> Fri Mar 02 04:57:25 CET 2018, data05022a.tar.2.bak
>> Mon Mar 05 04:45:18 CET 2018, data05262a.tar
>> Mon Mar 05 04:30:20 CET 2018, data05235a.tar
>> Mon Mar 05 04:44:44 CET 2018, data05259a.tar
>> Mon Mar 05 04:51:43 CET 2018, data05286a.tar
>> Mon Mar 05 04:54:03 CET 2018, journal.log
>> Fri Mar 02 04:57:24 CET 2018, data05022a.tar.bak
>> Mon Mar 05 04:32:32 CET 2018, data05238a.tar
>> Mon Mar 05 04:28:18 CET 2018, data05232a.tar
>> Mon Mar 05 04:45:07 CET 2018, data05261a.tar
>> Mon Mar 05 04:47:38 CET 2018, data05271a.tar
>> Mon Mar 05 04:49:48 CET 2018, data05279a.tar
>> Mon Mar 05 04:53:56 CET 2018, manifest
>> Mon Mar 05 04:34:30 CET 2018, data05241a.tar
>> Mon Mar 05 04:46:20 CET 2018, data05266a.tar
>> Mon Mar 05 04:43:41 CET 2018, data05255a.tar
>> Mon Mar 05 04:38:21 CET 2018, data05246a.tar
>> Mon Mar 05 04:48:14 CET 2018, data05273a.tar
>> Mon Mar 05 04:52:50 CET 2018, data05289a.tar
>> Mon Mar 05 04:42:19 CET 2018, data05252a.tar
>> Mon Mar 05 04:41:47 CET 2018, data05251a.tar
>> Mon Mar 05 04:47:07 CET 2018, data05269a.tar
>> Mon Mar 05 04:50:34 CET 2018, data05282a.tar
>> Mon Mar 05 04:47:21 CET 2018, data05270a.tar
>> Mon Mar 05 04:39:50 CET 2018, data05248a.tar
>> Mon Mar 05 04:29:40 CET 2018, data05234a.tar
>> Mon Mar 05 04:44:19 CET 2018, data05257a.tar
>> Mon Mar 05 04:44:30 CET 2018, data05258a.tar
>> Mon Mar 05 04:32:13 CET 2018, data05237a.tar
>> Mon Mar 05 04:48:43 CET 2018, data05275a.tar
>> Mon Mar 05 04:50:05 CET 2018, data05280a.tar
>> Mon Mar 05 04:37:30 CET 2018, data05245a.tar
>> Mon Mar 05 04:51:02 CET 2018, data05284a.tar
>> Mon Mar 05 04:46:33 CET 2018, data05267a.tar
>> Mon Mar 05 04:54:03 CET 2018, gc.log
>> size 16.1 GB (16081145586 bytes)
>> removed files [data05228a.tar, data05222a.tar, data05167a.tar,
>> data05184a.tar, data05214a.tar, data05192a.tar, data05200a.tar,
>> data05198a.tar, data05183a.tar, data05189a.tar, data05205a.tar,
>> data05175a.tar, data05230a.tar, data05206a.tar, data05191a.tar,
>> data05197a.tar, data05174a.tar, data05177a.tar, data05188a.tar,
>> data05171a.tar, data05182a.tar, data05169a.tar, data05224a.tar,
>> data05227a.tar, data05221a.tar, data05213a.tar, data05193a.tar,
>> data05185a.tar, data05190a.tar, data05216a.tar, data05219a.tar,
>> data05196a.tar, data05199a.tar, data05195a.tar, data05211a.tar,
>> data05187a.tar, data05203a.tar, data05208a.tar, data05186a.tar,
>> data05172a.tar, data05180a.tar, data05202a.tar, data05210a.tar,
>> data05178a.tar, data05209a.tar, data05217a.tar, data05225a.tar,
>> data05194a.tar, data05207a.tar, data05218a.tar, data05229a.tar,
>> data05220a.tar, data05168a.tar, data05223a.tar, data05226a.tar,
>> data05179a.tar, data05176a.tar, data05231a.tar, data05170a.tar,
>> data05215a.tar, data05201a.tar, data05181a.tar, data05173a.tar,
>> data05204a.tar, data05212a.tar]
>> added files [data05259a.tar, data05267a.tar, data05245a.tar,
>> data05237a.tar, data05253a.tar, data05261a.tar, data05283a.tar,
>> data05244a.tar, data05258a.tar, data05291a.tar, data05236a.tar,
>> data05275a.tar, data05260a.tar, data05252a.tar, data05289a.tar,
>> data05232a.tar, data05246a.tar, data05243a.tar, data05238a.tar,
>> data05235a.tar, data05240a.tar, data05279a.tar, data05262a.tar,
>> data05268a.tar, data05276a.tar, data05251a.tar, data05284a.tar,
>> data05270a.tar, data05254a.tar, data05273a.tar, data05281a.tar,
>> data05265a.tar, data05249a.tar, data05257a.tar, data05250a.tar,
>> data05256a.tar, data05264a.tar, data05239a.tar, data05287a.tar,
>> data05272a.tar, data05242a.tar, data05248a.tar, data05241a.tar,
>> data05280a.tar, data05247a.tar, data05255a.tar, data05278a.tar,
>> data05233a.tar, data05286a.tar, data05271a.tar, data05274a.tar,
>> data05288a.tar, data05263a.tar, data05266a.tar, data05285a.tar,
>> data05290a.tar, data05277a.tar, data05282a.tar, data05269a.tar,
>> data05234a.tar]
>> Compaction succeeded in 27.29 min (1637s).
>>
>> ```
>>
>> Greets,
>> Roy
signature.asc
Description: Message signed with OpenPGP
