[
https://issues.apache.org/jira/browse/JCR-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13922040#comment-13922040
]
Shashank Gupta edited comment on JCR-3731 at 3/6/14 5:36 AM:
-------------------------------------------------------------
Attached patched.
Created thread pool to migrate files from file system to S3. For
570 MB initial data, time reduced to 70s from 307s.
Result:
h5. Migration using 1 thread:
{code}
18.12.2013 05:08:36.877 *INFO* [FelixStartLevel]
org.apache.jackrabbit.core.data.CachingDataStore Uploaded: {3875}/{3875}
files,
{570535403}/{570535403} size data, time taken {307} sec
{code}
h5. Migration using 10 threads:
{code}
18.12.2013 04:56:52.176 *INFO* [FelixStartLevel]
org.apache.jackrabbit.core.data.CachingDataStore Uploaded: {3875}/{3875}
files,
{570535403}/{570535403} size data, time taken {70} sec
{code}
was (Author: shgupta):
Attached patched.
Created thread pool to migrate files from file system to S3. For
570 MB initial data, time reduced to 70s from 307s.
Result:
h5. Migration using 1 thread:
{code}
18.12.2013 05:08:36.877 *INFO* [FelixStartLevel]
com.day.crx.cloud.s3.ds.CachingDataStore Uploaded: {3875}/{3875} files,
{570535403}/{570535403} size data, time taken {307} sec
{code}
> Multi-threaded migration of binary files from FileSystem to S3 datastore
> -------------------------------------------------------------------------
>
> Key: JCR-3731
> URL: https://issues.apache.org/jira/browse/JCR-3731
> Project: Jackrabbit Content Repository
> Issue Type: Sub-task
> Components: jackrabbit-core
> Affects Versions: 2.7.4
> Reporter: Shashank Gupta
> Fix For: 2.7.5
>
> Attachments: JCR-3651-svn-updated.patch
>
>
> As per today, when we switch repository from FileDataStore to S3DataStore all
> binary files are migrated from local file system to S3Datastore. As per today
> this process is single threaded and takes lot of time. For e.g. for 1G intial
> content, it takes around 5 min to migrated from ec2 instance to S3.
> It can be made faster by migrating content in multi-threaded environment.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
