Hi,
If you want to do a mass import, please take a look at Nuxeo-Shell and
the fsimport / mtfsimport commands.
On an average server, document creation rate is between 40 and 60 docs/s.
About your problem :
If you do a loop 600 times that creates documents and only do the save
at the end :
- you do a big transaction that commits 600 documents
=> this commit can be very long and consume a lot of memory
- your folders won't be visible until the save is reached
=> until the commit is done, nothing is visible to other sessions
- when the session save is done, it triggers reindexing for the 600
documents
=> this can be resources consuming too
Change your code to do commit in batch :
=> you will have better performances and less resources used
=> you will be able to see your data while importing
Also, during batch insertion, we usually deactivate indexing : it's
faster to batch insert and then batch index.
for example, if you want to disable fulltext async indexing :
...
Map<String,Object> options = new HashMap<String, Object>();
options.put("BLOCK_JMS_PRODUCING", true);
String docType = "Folder";
DocumentModel doc = session.createDocumentModel(docType, options);
doc.setPathInfo(parent.getPathAsString(), name);
doc.setProperty("dublincore", "title", file.getName());
if (blockJMS) {
doc.putContextData("BLOCK_JMS_PRODUCING", true);
}
if (blockSyncIndexing) {
doc.putContextData("BLOCK_SYNC_INDEXING", true); // if you want to
block sync indexing
}
doc = session.createDocument(doc);
...
NB : If you block sync indexing, you won't be able to see your folders
in the UI until you reindex since the default nav used search service ...
[EMAIL PROTECTED] a écrit :
> Hi all,
>
> I have to migrate more than 300,000 documents from a custom ECM to Nuxeo EP.
> I needed to develop a client application to do the work. It looks to me that
> class SampleApplication from nuxeo-api-sample was the thing I needed. So I
> started my own class to create first the folders tree (I have to create more
> than 600 folders on 5 levels). here is the (pseudo) code I used to create
> them :
>
> RepositoryManager repositoryMgr =
> Framework.getService(RepositoryManager.class);
> Repository repository = repositoryMgr.getRepository(repName);
> Framework.login("Administrator", "Administrator");
> CoreSession remote = repository.open();
>
> // repeat 600 times ...
> {
> DocumentModel dm = new DocumentModelImpl(parentDocModel, name, "Folder");
> remote.createDocument(dm);
> }
> remote.save();
>
> I have 3 problems :
> 1- it took more than 1 hour to create the 600 folders !
> 2- I was not able to browse the newly created folders during the creation
> procedure.
> 3- At the end of the procedure java.exe was using more than 1 GB.
>
> I am very worried about the performance, 1 hour for 600 folders, it's going
> to take many hours to create 300,000 documents.
> Maybe I am doing something wrong, or I forgot something, or I am not using a
> good way ti import files in Nuxeo EP.
>
> Can someone give me some help ?
>
>
> Kr.
> _______________________________________________
> ECM mailing list
> [email protected]
> http://lists.nuxeo.com/mailman/listinfo/ecm
>
>
_______________________________________________
ECM mailing list
[email protected]
http://lists.nuxeo.com/mailman/listinfo/ecm
