[jira] [Updated] (JCR-3731) Multi-threaded migration of binary files from FileSystem to S3 datastore
[ https://issues.apache.org/jira/browse/JCR-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominique Pfister updated JCR-3731: --- Resolution: Fixed Status: Resolved (was: Patch Available) Patch revised and committed in revision 1577127 to trunk. Multi-threaded migration of binary files from FileSystem to S3 datastore - Key: JCR-3731 URL: https://issues.apache.org/jira/browse/JCR-3731 Project: Jackrabbit Content Repository Issue Type: Sub-task Components: jackrabbit-core Affects Versions: 2.7.4 Reporter: Shashank Gupta Assignee: Dominique Pfister Fix For: 2.7.5 Attachments: JCR-3651.patch As per today, when we switch repository from FileDataStore to S3DataStore all binary files are migrated from local file system to S3Datastore. As per today this process is single threaded and takes lot of time. For e.g. for 1G intial content, it takes around 5 min to migrated from ec2 instance to S3. It can be made faster by migrating content in multi-threaded environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (JCR-3731) Multi-threaded migration of binary files from FileSystem to S3 datastore
[
https://issues.apache.org/jira/browse/JCR-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashank Gupta updated JCR-3731:
Attachment: JCR-3651-svn-updated.patch
Attached patched.
Created thread pool to migrate files from file system to S3. For
570 MB initial data, time reduced to 70s from 307s.
Result:
h5. Migration using 1 thread:
{code}
18.12.2013 05:08:36.877 *INFO* [FelixStartLevel]
com.day.crx.cloud.s3.ds.CachingDataStore Uploaded: {3875}/{3875} files,
{570535403}/{570535403} size data, time taken {307} sec
{code}
Multi-threaded migration of binary files from FileSystem to S3 datastore
-
Key: JCR-3731
URL: https://issues.apache.org/jira/browse/JCR-3731
Project: Jackrabbit Content Repository
Issue Type: Sub-task
Components: jackrabbit-core
Affects Versions: 2.7.4
Reporter: Shashank Gupta
Fix For: 2.7.5
Attachments: JCR-3651-svn-updated.patch
As per today, when we switch repository from FileDataStore to S3DataStore all
binary files are migrated from local file system to S3Datastore. As per today
this process is single threaded and takes lot of time. For e.g. for 1G intial
content, it takes around 5 min to migrated from ec2 instance to S3.
It can be made faster by migrating content in multi-threaded environment.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (JCR-3731) Multi-threaded migration of binary files from FileSystem to S3 datastore
[ https://issues.apache.org/jira/browse/JCR-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashank Gupta updated JCR-3731: Status: Patch Available (was: Open) Hi [~tmueller] please review the patch. Thanks. Multi-threaded migration of binary files from FileSystem to S3 datastore - Key: JCR-3731 URL: https://issues.apache.org/jira/browse/JCR-3731 Project: Jackrabbit Content Repository Issue Type: Sub-task Components: jackrabbit-core Affects Versions: 2.7.4 Reporter: Shashank Gupta Fix For: 2.7.5 Attachments: JCR-3651-svn-updated.patch As per today, when we switch repository from FileDataStore to S3DataStore all binary files are migrated from local file system to S3Datastore. As per today this process is single threaded and takes lot of time. For e.g. for 1G intial content, it takes around 5 min to migrated from ec2 instance to S3. It can be made faster by migrating content in multi-threaded environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (JCR-3731) Multi-threaded migration of binary files from FileSystem to S3 datastore
[ https://issues.apache.org/jira/browse/JCR-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashank Gupta updated JCR-3731: Attachment: JCR-3651.patch Multi-threaded migration of binary files from FileSystem to S3 datastore - Key: JCR-3731 URL: https://issues.apache.org/jira/browse/JCR-3731 Project: Jackrabbit Content Repository Issue Type: Sub-task Components: jackrabbit-core Affects Versions: 2.7.4 Reporter: Shashank Gupta Fix For: 2.7.5 Attachments: JCR-3651.patch As per today, when we switch repository from FileDataStore to S3DataStore all binary files are migrated from local file system to S3Datastore. As per today this process is single threaded and takes lot of time. For e.g. for 1G intial content, it takes around 5 min to migrated from ec2 instance to S3. It can be made faster by migrating content in multi-threaded environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (JCR-3731) Multi-threaded migration of binary files from FileSystem to S3 datastore
[ https://issues.apache.org/jira/browse/JCR-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashank Gupta updated JCR-3731: Attachment: (was: JCR-3651-svn-updated.patch) Multi-threaded migration of binary files from FileSystem to S3 datastore - Key: JCR-3731 URL: https://issues.apache.org/jira/browse/JCR-3731 Project: Jackrabbit Content Repository Issue Type: Sub-task Components: jackrabbit-core Affects Versions: 2.7.4 Reporter: Shashank Gupta Fix For: 2.7.5 Attachments: JCR-3651.patch As per today, when we switch repository from FileDataStore to S3DataStore all binary files are migrated from local file system to S3Datastore. As per today this process is single threaded and takes lot of time. For e.g. for 1G intial content, it takes around 5 min to migrated from ec2 instance to S3. It can be made faster by migrating content in multi-threaded environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (JCR-3731) Multi-threaded migration of binary files from FileSystem to S3 datastore
[ https://issues.apache.org/jira/browse/JCR-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashank Gupta updated JCR-3731: Description: As per today, when we switch repository from FileDataStore to S3DataStore all binary files are migrated from local file system to S3Datastore. As per today this process is single threaded and takes lot of time. For e.g. for 1G intial content, it takes around 5 min to migrated from ec2 instance to S3. It can be made faster by migrating content in multi-threaded environment. was: As per today, when we switch CRX from ClusterDataStore to S3DataStore all binary files are migrated from local file system to S3Datastore. As per today this process is single threaded and takes lot of time. For e.g. for 1G intial content, it takes around 5 min to migrated from ec2 instance to S3. It can be made faster by migrating content in multi-threaded environment. Multi-threaded migration of binary files from FileSystem to S3 datastore - Key: JCR-3731 URL: https://issues.apache.org/jira/browse/JCR-3731 Project: Jackrabbit Content Repository Issue Type: Sub-task Components: jackrabbit-core Affects Versions: 2.7.4 Reporter: Shashank Gupta Fix For: 2.7.5 As per today, when we switch repository from FileDataStore to S3DataStore all binary files are migrated from local file system to S3Datastore. As per today this process is single threaded and takes lot of time. For e.g. for 1G intial content, it takes around 5 min to migrated from ec2 instance to S3. It can be made faster by migrating content in multi-threaded environment. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
