[ 
https://issues.apache.org/jira/browse/OAK-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119568#comment-16119568
 ] 

Andrei Dulceanu commented on OAK-5902:
--------------------------------------

[~frm], [~mduerig]

bq. Before committing, the problems with the memory consumption in 
{{DataStoreTestBase.testSyncBigBlog}} 
I think the memory consumption is the key here. In an investigation from some 
time ago, 4GB of heap were needed for 1GB blob and 6GB for 2GB blob. This was 
in part due to using {{addTestContent}} in the investigation, which allocates 
that huge {{byte[]}} on the heap. With the new approach in 
{{addTestContentOnTheFly}} this problem is solved, and the chunking per se 
improved things a lot. We are now in the position of successfully syncing 2.5 
GB blob with only 3.5 GB memory. 

bq. the running time in ExternalPrivateStoreIT should be investigated.
My analysis shows that {{51s}} are spent for adding the test content (i.e. 2.5 
GB blob), {{61s}} are spent for syncing between master and standby and another 
{{44s}} are spent for checking that the sync was ok (i.e. comparing two streams 
summing up to 2.5 GB). I find nothing unusual here.

bq. Agreed, increasing the heap for the tests is problematic and we shouldn't 
do this. At least we need to understand where the memory requirements come 
from: is it the test or the code?
Agree. I suggest to create a separate issue for analysing the memory 
consumption and to commit all the changes, except:
* heap size increase in {{pom.xml}}
* annotate {{testSyncBigBlog}} with {{@Ignore(OAK-XXX)}}

Since all our ITs for cold standby use chunking now (the default {{1MB}} chunk 
size) and they all pass, I'd say we can safely commit the rest of the changes, 
as explained above.

WDYT?



> Cold standby should allow syncing of blobs bigger than 2.2 GB
> -------------------------------------------------------------
>
>                 Key: OAK-5902
>                 URL: https://issues.apache.org/jira/browse/OAK-5902
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: segment-tar
>    Affects Versions: 1.6.1
>            Reporter: Andrei Dulceanu
>            Assignee: Andrei Dulceanu
>            Priority: Minor
>             Fix For: 1.8, 1.7.6
>
>
> Currently there is a limitation for the maximum binary size (in bytes) to be 
> synced between primary and standby instances. This matches 
> {{Integer.MAX_VALUE}} (2,147,483,647) bytes and no binaries bigger than this 
> limit can be synced between the instances.
> Per comment at [1], the current protocol needs to be changed to allow sending 
> of binaries in chunks, to surpass this limitation.
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/standby/client/StandbyClient.java#L125



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to