[
https://issues.apache.org/jira/browse/OAK-7027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278670#comment-16278670
]
Francesco Mari commented on OAK-7027:
-------------------------------------
The test seems to be failing because the communication between the primary and
the standby is faster than expected. The read timeout is reset every time new
data flows from the primary to the standby. That is, if no data is sent from
the primary to the standby in the interval of time specified by the read
timeout, a {{ReadTimeoutException}} is thrown. When binaries were sent in one
response, it was easy to exceed the read timeout, and the test was not failing.
By sending binaries in chunks, the read timeout is more difficult to trigger.
The {{ReadTimeoutException}} seems to happen at the very end of the transfer
process, when the client is kept open while the standby instance processes the
binary. In this time span no further requests are performed, and no data flows
from the primary to standby. To prevent this from happening it is necessary to
close the client as soon as it's not needed.
> Test failure: ExternalPrivateStoreIT.testSyncFailingDueToTooShortTimeout
> ------------------------------------------------------------------------
>
> Key: OAK-7027
> URL: https://issues.apache.org/jira/browse/OAK-7027
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar, tarmk-standby
> Reporter: Michael Dürig
> Assignee: Francesco Mari
> Labels: test-failure
> Fix For: 1.8, 1.7.13
>
>
> Seen on an internal Windows Jenkins node:
> h3. Regression
> org.apache.jackrabbit.oak.segment.standby.ExternalPrivateStoreIT.testSyncFailingDueToTooShortTimeout
> h3. Error Message
> {noformat}
> Values should be different. Actual: { root = { ... } }
> {noformat}
> h3. Stacktrace
> {noformat}
> java.lang.AssertionError: Values should be different. Actual: { root = { ...
> } }
> at
> org.apache.jackrabbit.oak.segment.standby.ExternalPrivateStoreIT.testSyncFailingDueToTooShortTimeout(ExternalPrivateStoreIT.java:87)
> {noformat}
> h3. Standard Output
> {noformat}
> 22:41:13.646 INFO [main] FileStoreBuilder.java:340 Creating file
> store FileStoreBuilder{version=1.8-SNAPSHOT,
> directory=target\junit2834122541179880349\junit3041268421527563090,
> blobStore=DataStore backed BlobStore
> [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1,
> segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0,
> stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000,
> nodeDeduplicationCacheSize=1, memoryMapping=false,
> gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false,
> gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60,
> retainedGenerations=2, gcType=FULL}}
> 22:41:13.646 INFO [main] FileStore.java:241 TarMK opened at
> target\junit2834122541179880349\junit3041268421527563090, mmap=false, size=0
> B (0 bytes)
> 22:41:13.646 DEBUG [main] FileStore.java:247 TAR files:
> TarFiles{readers=[],writer=target\junit2834122541179880349\junit3041268421527563090\data00000a.tar}
> 22:41:13.646 DEBUG [main] TarWriter.java:185 Writing segment
> 4cea1684-ef05-44f5-a869-3ef2df6e0c9a to
> target\junit2834122541179880349\junit3041268421527563090\data00000a.tar
> 22:41:13.646 INFO [main] FileStoreBuilder.java:340 Creating file
> store FileStoreBuilder{version=1.8-SNAPSHOT,
> directory=target\junit2834122541179880349\junit4470899745425503556,
> blobStore=DataStore backed BlobStore
> [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1,
> segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0,
> stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000,
> nodeDeduplicationCacheSize=1, memoryMapping=false,
> gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false,
> gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60,
> retainedGenerations=2, gcType=FULL}}
> 22:41:13.646 INFO [main] FileStore.java:241 TarMK opened at
> target\junit2834122541179880349\junit4470899745425503556, mmap=false, size=0
> B (0 bytes)
> 22:41:13.646 DEBUG [main] FileStore.java:247 TAR files:
> TarFiles{readers=[],writer=target\junit2834122541179880349\junit4470899745425503556\data00000a.tar}
> 22:41:13.646 DEBUG [main] TarWriter.java:185 Writing segment
> 8d19c7dc-8b48-4e10-a58d-31c15c93f2fe to
> target\junit2834122541179880349\junit4470899745425503556\data00000a.tar
> 22:41:13.646 INFO [main] DataStoreTestBase.java:127 Test begin:
> testSyncFailingDueToTooShortTimeout
> 22:41:13.646 INFO [main] SegmentNodeStore.java:120 Creating segment
> node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore
> [org.apache.jackrabbit.core.data.FileDataStore]}
> 22:41:13.646 INFO [main] LockBasedScheduler.java:155 Initializing
> SegmentNodeStore with the commitFairLock option enabled.
> 22:41:13.708 DEBUG [main] StandbyServer.java:248 Binding was
> successful
> 22:41:13.708 DEBUG [main] TarWriter.java:185 Writing segment
> 4a5183bd-bcdf-41ab-a557-6f19143bbc91 to
> target\junit2834122541179880349\junit3041268421527563090\data00000a.tar
> 22:41:13.739 DEBUG [main] TarRevisions.java:240 TarMK journal
> update null -> 4a5183bd-bcdf-41ab-a557-6f19143bbc91.0000000c
> 22:41:13.755 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request
> from client 9aa63ed8-347b-4f00-ae7c-f984e0623e90 for current head
> 22:41:13.755 DEBUG [primary-1] ClientFilterHandler.java:53 Client
> /127.0.0.1:65480 is allowed
> 22:41:13.755 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head'
> message
> 22:41:13.755 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get
> head' received from client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.755 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for
> client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.755 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head
> 4a5183bd-bcdf-41ab-a557-6f19143bbc91.0000000c to client
> 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.755 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get
> head' response
> 22:41:13.755 DEBUG [standby-run-23] StandbyClientSyncExecution.java:103 Found
> missing segment 4a5183bd-bcdf-41ab-a557-6f19143bbc91
> 22:41:13.755 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124
> Inspecting segment 4a5183bd-bcdf-41ab-a557-6f19143bbc91
> 22:41:13.755 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending
> request from client 9aa63ed8-347b-4f00-ae7c-f984e0623e90 for references of
> segment 4a5183bd-bcdf-41ab-a557-6f19143bbc91
> 22:41:13.755 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get
> references' message
> 22:41:13.771 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading
> references of segment 4a5183bd-bcdf-41ab-a557-6f19143bbc91 for client
> 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending
> references of segment 4a5183bd-bcdf-41ab-a557-6f19143bbc91 to client
> 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get
> references' response
> 22:41:13.771 DEBUG [standby-run-23] StandbyClientSyncExecution.java:184 Found
> reference from 4a5183bd-bcdf-41ab-a557-6f19143bbc91 to
> 4cea1684-ef05-44f5-a869-3ef2df6e0c9a
> 22:41:13.771 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124
> Inspecting segment 4cea1684-ef05-44f5-a869-3ef2df6e0c9a
> 22:41:13.771 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending
> request from client 9aa63ed8-347b-4f00-ae7c-f984e0623e90 for references of
> segment 4cea1684-ef05-44f5-a869-3ef2df6e0c9a
> 22:41:13.771 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get
> references' message
> 22:41:13.771 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading
> references of segment 4cea1684-ef05-44f5-a869-3ef2df6e0c9a for client
> 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending
> references of segment 4cea1684-ef05-44f5-a869-3ef2df6e0c9a to client
> 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get
> references' response
> 22:41:13.771 INFO [standby-run-23] StandbyClientSyncExecution.java:196
> Copying data segment 4cea1684-ef05-44f5-a869-3ef2df6e0c9a from primary
> 22:41:13.771 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending
> request from client 9aa63ed8-347b-4f00-ae7c-f984e0623e90 for segment
> 4cea1684-ef05-44f5-a869-3ef2df6e0c9a
> 22:41:13.771 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get
> segment' message
> 22:41:13.771 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get
> segment' received from client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading
> segment 4cea1684-ef05-44f5-a869-3ef2df6e0c9a for client
> 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [primary-1] CommunicationObserver.java:125 Segment with
> size 192 sent to client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending
> segment 4cea1684-ef05-44f5-a869-3ef2df6e0c9a to client
> 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get
> segment' response
> 22:41:13.771 DEBUG [standby-run-23] TarWriter.java:185 Writing segment
> 4cea1684-ef05-44f5-a869-3ef2df6e0c9a to
> target\junit2834122541179880349\junit4470899745425503556\data00000a.tar
> 22:41:13.771 INFO [standby-run-23] StandbyClientSyncExecution.java:196
> Copying data segment 4a5183bd-bcdf-41ab-a557-6f19143bbc91 from primary
> 22:41:13.771 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending
> request from client 9aa63ed8-347b-4f00-ae7c-f984e0623e90 for segment
> 4a5183bd-bcdf-41ab-a557-6f19143bbc91
> 22:41:13.771 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get
> segment' message
> 22:41:13.771 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get
> segment' received from client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading
> segment 4a5183bd-bcdf-41ab-a557-6f19143bbc91 for client
> 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [primary-1] CommunicationObserver.java:125 Segment with
> size 448 sent to client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending
> segment 4a5183bd-bcdf-41ab-a557-6f19143bbc91 to client
> 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get
> segment' response
> 22:41:13.771 DEBUG [standby-run-23] TarWriter.java:185 Writing segment
> 4a5183bd-bcdf-41ab-a557-6f19143bbc91 to
> target\junit2834122541179880349\junit4470899745425503556\data00000a.tar
> 22:41:13.771 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request
> from client 9aa63ed8-347b-4f00-ae7c-f984e0623e90 for blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880
> 22:41:13.771 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob'
> request
> 22:41:13.771 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get
> blob id' received from client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880 for
> client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [primary-1] CommunicationObserver.java:130 Binary with
> size 5242880 sent to client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.771 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880 to
> client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.786 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5
> of size 1048576 from blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880 to
> client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.802 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5
> of size 1048576 from blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880 to
> client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.802 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get
> blob' response
> 22:41:13.802 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk
> 1/5 of size 1048576 from blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880
> 22:41:13.802 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK.
> Appending chunk to disk to
> C:\Windows\TEMP\c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880.tmp
>
> 22:41:13.802 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5
> of size 1048576 from blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880 to
> client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.802 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get
> blob' response
> 22:41:13.818 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk
> 2/5 of size 1048576 from blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880
> 22:41:13.818 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK.
> Appending chunk to disk to
> C:\Windows\TEMP\c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880.tmp
>
> 22:41:13.818 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5
> of size 1048576 from blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880 to
> client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.818 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get
> blob' response
> 22:41:13.818 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk
> 3/5 of size 1048576 from blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880
> 22:41:13.818 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK.
> Appending chunk to disk to
> C:\Windows\TEMP\c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880.tmp
>
> 22:41:13.818 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5
> of size 1048576 from blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880 to
> client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> 22:41:13.833 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get
> blob' response
> 22:41:13.833 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk
> 4/5 of size 1048576 from blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880
> 22:41:13.833 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK.
> Appending chunk to disk to
> C:\Windows\TEMP\c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880.tmp
>
> 22:41:13.833 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get
> blob' response
> 22:41:13.833 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk
> 5/5 of size 1048576 from blob
> c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880
> 22:41:13.833 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK.
> Appending chunk to disk to
> C:\Windows\TEMP\c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880.tmp
>
> 22:41:13.833 DEBUG [standby-1] ResponseDecoder.java:167 Received entire
> blob c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880
> 22:41:13.880 DEBUG [standby-run-23] ResponseDecoder.java:66 Processing input
> stream finished! Deleting file
> C:\Windows\TEMP\c3ac16ad0c8bc3a3ff30cef8e296af92d53058c13a8930406d3f08e271b4b57b#5242880.tmp
> 22:41:13.896 DEBUG [standby-run-23] TarRevisions.java:240 TarMK journal
> update null -> 4a5183bd-bcdf-41ab-a557-6f19143bbc91.0000000c
> 22:41:13.911 WARN [standby-1] ExceptionHandler.java:37 Exception caught
> on client 9aa63ed8-347b-4f00-ae7c-f984e0623e90
> io.netty.handler.timeout.ReadTimeoutException: null
> 22:41:13.911 INFO [standby-run-23] StandbyClientSyncExecution.java:82
> updated head state successfully: true in 156ms.
> 22:41:13.911 DEBUG [standby-run-23] StandbyClient.java:157 Channel closed
> 22:41:16.137 DEBUG [main] StandbyClientSync.java:277 Group shut down
> 22:41:16.137 DEBUG [main] StandbyServer.java:219 Channel
> disconnected
> 22:41:16.137 DEBUG [main] StandbyServer.java:219 Channel
> disconnected
> 22:41:16.137 DEBUG [main] StandbyServer.java:230 Boss group shut
> down
> 22:41:16.137 DEBUG [main] StandbyServer.java:236 Worker group shut
> down
> 22:41:16.137 INFO [main] DataStoreTestBase.java:132 Test end:
> testSyncFailingDueToTooShortTimeout
> 22:41:16.137 DEBUG [main] Scheduler.java:134 The scheduler
> FileStore background tasks was successfully shut down
> 22:41:16.137 DEBUG [main] TarRevisions.java:236 Head state did
> not change, skipping flush
> 22:41:16.184 INFO [main] FileStore.java:480 TarMK closed:
> target\junit2834122541179880349\junit4470899745425503556
> 22:41:16.184 DEBUG [main] Scheduler.java:134 The scheduler
> FileStore background tasks was successfully shut down
> 22:41:16.184 DEBUG [main] TarRevisions.java:236 Head state did
> not change, skipping flush
> 22:41:16.199 INFO [main] FileStore.java:480 TarMK closed:
> target\junit2834122541179880349\junit3041268421527563090
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)