Yeah, you're right. I would suggest if it works with a 40.000 Users import/export, at lets say -Xmx512MB for the hole red5 server process, I think its an acceptable performance.
Sebastian 2011/7/21 Maxim Solodovnik <[email protected]>: > If memory used amount of memory will be unacceptably high we can use > temporary tables for store id pairs. > With openJPA it will be just couple additional beans. > > > On Thu, Jul 21, 2011 at 16:21, [email protected] <[email protected]> > wrote: >> >> Hi Maxim, >> >> yes this was the thing that I tried to avoid :) Collecting all IDs and >> building lookup tables to see which old id corresponds to the record >> after the import. >> >> The other thing, to import the objects in the correct order is already >> implemented. >> >> Some Issue will be also the uploaded files: There are some >> restrictions on the user's profile images. I think they have the >> userId as folderName in the path. So you can't copy the folders >> anymore into the disk just by coping the ROOT folder of the >> upload-directory from the Backup-ZIP to the disk, cause there are some >> subfolders that contain invalid folderNames that need to be >> rewritten/renamed. >> >> We can try that out, its obviously better if it works but its a rather >> big change. >> My concern is the same like yours: Could be quite memory hungry process. >> >> I got for example a database with 40.000++ users that I need to >> import/export. Holding all 40.000 users would not work, maybe holding >> the 40.000 IDs could work. >> >> Sebastian >> >> 2011/7/21 Maxim Solodovnik <[email protected]>: >> > Hello Sebastian, Vasiliy >> > Instead of creating generator classes I would like to propose the >> > following >> > mechanism: >> > 1) Import should be done in "well known" order i.e. organisanisations >> > are >> > imported first then users, then rooms etc. >> > 2) While importing certain object id is not set. Instead of this old id >> > and >> > new id are stored in hashtable key== oldId, value == newId >> > 3) While importing objects containing links to "parent objects" newId is >> > used using Hashtable created. >> > 4) Pictures/streams/videos are stored in the file system using newId >> > The import will be memory consuming operation but all constraints will >> > be >> > valid. >> > >> > On Mon, Jul 18, 2011 at 17:46, Vasya <[email protected]> wrote: >> >> >> >> Hi Sebastian, >> >> >> >> Thanks for your answer! >> >> >> >> I will continue to investigate problem tomorrow. >> >> >> >> Vasiliy. >> >> >> >> >> >> On Jul 18, 5:32 pm, "[email protected]" <[email protected]> >> >> wrote: >> >> > Hi Vasya, >> >> > >> >> > If tested getBackupRooms yesterday, it was okay for me. >> >> > >> >> > The thing with the *deleted* is that it has to return ALL records, no >> >> > matter if the deleted flag is set or not. >> >> > This is the case because of the foreign key references. >> >> > >> >> > Just imagine the following sample: >> >> > Rooms-Table: >> >> > ID name deleted >> >> > 1 room1 false >> >> > 2 room2 true >> >> > 3 room3 false >> >> > >> >> > Rooms-Organizations Table: >> >> > ID rooms_id organisation_id >> >> > 1 1 1 >> >> > 1 2 2 >> >> > 1 3 1 >> >> > >> >> > If you only export those records that are deleted == 'false' >> >> > => the importer would add room3 as the record with the ID 2 and as >> >> > consequence the room would be assigned to the organization 2 => which >> >> > is wrong of course. >> >> > >> >> > And the same you have between all tables. That is why I did import >> >> > records always WITH all records marked as "deleted" AND in the import >> >> > process I did add each object with the ID ALREADY SET. That means I >> >> > did not use the generated Id from Hibernate, I did use the ID that >> >> > was >> >> > provided with the XML. But this does not work anymore as openJPA does >> >> > not accept objects where the ID is > 0. >> >> > At the moment everything is fine if you do a single import directly >> >> > after the installation. But for example a user creates an >> >> > Organization >> >> > (will be Id = 2), deletes it again, the ID=2 is gone, if we do any >> >> > import now with references to an organization=2 => everything Fails. >> >> > In the old implementation we still where able to add an Org with ID=2 >> >> > cause we did provide the ID in the Import Object and did not use the >> >> > generated one for objects that are created using the import. >> >> > >> >> > That is the problem, if you have an idea on what we might can do >> >> > about >> >> > those ID problem, maybe there is a Generator Class in openJPA that >> >> > accepts self generated IDs. >> >> > >> >> > Sebastian >> >> > >> >> > 2011/7/18 Vasya <[email protected]>: >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > > Hi Sebastian, >> >> > >> >> > > I have found and fixed two errors in backup: >> >> > > 1. Roommanagement.getInstance().getBackupRooms() returns error >> >> > > so >> >> > > root >> >> > > condition of criteria query is not defined. >> >> > > 2. BackupImport.getAppointmentListByXML method does not work >> >> > > properly >> >> > > because ‘user_id’ is not set. >> >> > > I have fixed these errors and sent you email with patch. >> >> > >> >> > > So, I have question about >> >> > > UsersDaoImpl.getInstance().getAllUsersDeleted() method. It’s not >> >> > > cleared for me what list of users should be returns all users or >> >> > > deleted users? This method calls from BackupExport for export >> >> > > users. >> >> > > Please clarify what list of users should be return from this >> >> > > method. >> >> > >> >> > > Tomorrow I will continue to work on openmeetings and try to resolve >> >> > > other problems. >> >> > >> >> > > Thanks, >> >> > > Vasiliy. >> >> > >> >> > > On Jul 18, 1:55 pm, "[email protected]" <[email protected]> >> >> > > wrote: >> >> > >> Hi Vasya, >> >> > >> >> > >> I think I've fixed it, but some more testing would be good. >> >> > >> I also found out that sometimes emails contain the char:  (and >> >> > >>  and  ) >> >> > >> ... the SaxParser fails to read those files >> >> > >> ... I think it is because people copy/paste their email from >> >> > >> somewhere >> >> > >> in the fields. >> >> > >> But I have no idea about how they do it ... However this only >> >> > >> happens >> >> > >> 1 time per 1000 users but still strange. >> >> > >> >> > >> For the Primary Key Issue I have no idea too, I changed it now but >> >> > >> as >> >> > >> you can imagine it can get quite confusing if the system generates >> >> > >> the >> >> > >> id itself and you import xxx records that "should" reference each >> >> > >> other based on those IDs. For example rooms VS organisations or >> >> > >> users >> >> > >> VS organizations >> >> > >> >> > >> Sebastian >> >> > >> >> > >> 2011/7/18 Vasya <[email protected]>: >> >> > >> >> > >> > Hello Sebastian, >> >> > >> >> > >> > May be my test of backup was not completely. Today a have try to >> >> > >> > retest the import of backup more details. >> >> > >> >> > >> > Thanks, >> >> > >> >> > >> > Vasily. >> >> > >> >> > >> > On Jul 18, 1:12 am, "[email protected]" >> >> > >> > <[email protected]> >> >> > >> > wrote: >> >> > >> >> The import of Backup did not work. OpenJPA does not accept to >> >> > >> >> add >> >> > >> >> a >> >> > >> >> new object where the Primary Key is already set. This is >> >> > >> >> neccessary >> >> > >> >> for importing existing Records from one database to another >> >> > >> >> because of >> >> > >> >> the foreign key relation-ships. >> >> > >> >> > >> >> I have changed the code for now so that it works again. (in >> >> > >> >> r3997) >> >> > >> >> However, there are still some issues to be solved. >> >> > >> >> > >> >> You might never run into that, the imported checks if the Id >> >> > >> >> does >> >> > >> >> exists before it adds a new record. So if you export a freshly >> >> > >> >> installed openmeetings and then import it again into the same >> >> > >> >> => >> >> > >> >> Zero >> >> > >> >> happens. >> >> > >> >> So to test the backup import you really need to generate some >> >> > >> >> users/conferencerooms/calendar-events/private messages and >> >> > >> >> contacts et >> >> > >> >> cetera and then trying to import that backup into a fresh >> >> > >> >> installed >> >> > >> >> openmeetings database. >> >> > >> >> > >> >> Sebastian >> >> > >> >> > >> >> 2011/7/16 [email protected] <[email protected]>: >> >> > >> >> > >> >> > the saluation thing is fixed too.... >> >> > >> >> > >> >> > Sebastian >> >> > >> >> > >> >> > 2011/7/16 Maxim Solodovnik <[email protected]>: >> >> > >> >> >> great to hear it :) >> >> > >> >> >> please let me know in case of any issues you will found. >> >> > >> >> > >> >> >> On Sat, Jul 16, 2011 at 20:27, [email protected] >> >> > >> >> >> <[email protected]> >> >> > >> >> >> wrote: >> >> > >> >> > >> >> >>> ok ... >> >> > >> >> > >> >> >>> I've fixed that issue with the order. Now I need to add >> >> > >> >> >>> also >> >> > >> >> >>> the where >> >> > >> >> >>> clause in the annotation. This is really nice, this manual >> >> > >> >> >>> sub-select >> >> > >> >> >>> for the main-menu always was a nasty hack because the >> >> > >> >> >>> x-doclet >> >> > >> >> >>> hbm.xml >> >> > >> >> >>> creation did not allow those sub-queries with >> >> > >> >> >>> where+orderby. >> >> > >> >> >>> ... this >> >> > >> >> >>> is also the reason why we have STRING flag *deleted* and >> >> > >> >> >>> NOT a >> >> > >> >> >>> boolean >> >> > >> >> >>> field *deleted*. In previous hibernate/xdoclet creation it >> >> > >> >> >>> was >> >> > >> >> >>> not >> >> > >> >> >>> possible to write a annotation with a boolean value as >> >> > >> >> >>> where >> >> > >> >> >>> clause ^^ >> >> > >> >> >>> At some point we might clean that up too. >> >> > >> >> > >> >> >>> The other one with the strang saluations_id / Primary Key >> >> > >> >> >>> is >> >> > >> >> >>> still to be >> >> > >> >> >>> done. >> >> > >> >> > >> >> >>> Sebastian >> >> > >> >> > >> >> >>> 2011/7/16 [email protected] <[email protected]>: >> >> > >> >> >>> > Hi Maxim, >> >> > >> >> > >> >> >>> > I am currently checking out the openJPA refactoring. I >> >> > >> >> >>> > like >> >> > >> >> >>> > it, looks >> >> > >> >> >>> > very nice and using annotations instead of XML files is >> >> > >> >> >>> > quite elegant. >> >> > >> >> > >> >> >>> > Maybe you have an idea about those things: >> >> > >> >> >>> > - The navigation items are ordered different. For example >> >> > >> >> >>> > by >> >> > >> >> >>> > default >> >> > >> >> >>> > you have the calendar instead of the dashboard now. And >> >> > >> >> >>> > in >> >> > >> >> >>> > the >> >> > >> >> >>> > Administration you have a different order too. You have >> >> > >> >> >>> > some >> >> > >> >> >>> > code to >> >> > >> >> >>> > do the one-to-many relation in your mapping. I think we >> >> > >> >> >>> > might remove >> >> > >> >> >>> > that and use java.util.List again, I think java.util.Set >> >> > >> >> >>> > is >> >> > >> >> >>> > not save >> >> > >> >> >>> > in meaning of the order of items. >> >> > >> >> >>> > - The table saluations start with the id 251 in my case. >> >> > >> >> >>> > Normally >> >> > >> >> >>> > record 1 should have salutations_id 1. That is necessary >> >> > >> >> >>> > as >> >> > >> >> >>> > the its >> >> > >> >> >>> > hardcoded in the client adminstration panel that "Mr" has >> >> > >> >> >>> > the ID 1. >> >> > >> >> >>> > You got an idea about that? >> >> > >> >> > >> >> >>> > Sebastian >> >> > >> >> > >> >> >>> > 2011/7/16 Maxim Solodovnik <[email protected]>: >> >> > >> >> >>> >> Hello Sebastian, >> >> > >> >> >>> >> I just checked: the last extension listed is png >> >> > >> >> >>> >> and it is available for selection. >> >> > >> >> > >> >> >>> >> On Sat, Jul 16, 2011 at 17:58, [email protected] >> >> > >> >> >>> >> <[email protected]> >> >> > >> >> >>> >> wrote: >> >> > >> >> > >> >> >>> >>> Have you verified that all extension are visible then >> >> > >> >> >>> >>> in >> >> > >> >> >>> >>> the windows >> >> > >> >> >>> >>> upload dialog? >> >> > >> >> >>> >>> In previous flash version after xx extension it just >> >> > >> >> >>> >>> cuts >> >> > >> >> >>> >>> off and the >> >> > >> >> >>> >>> rest was not visible or selectable at all. That was >> >> > >> >> >>> >>> weird. >> >> > >> >> >>> >>> That is why >> >> > >> >> >>> >>> there are multiple columns and possibilities to choose. >> >> > >> >> > >> >> >>> >>> Sebastian >> >> > >> >> > >> >> >>> >>> 2011/7/16 <[email protected]>: >> >> > >> >> >>> >>> > Revision: 3988 >> >> > >> >> >>> >>> > Author: solomax666 >> >> > >> >> >>> >>> > Date: Sat Jul 16 02:59:13 2011 >> >> > >> >> >>> >>> > Log: - "Allowed Files" should contain all >> >> > >> >> >>> >>> > extensions allowed >> >> > >> >> >>> >>> >> >> > >> >> >>> >>> > >> >> > >> >> >>> >>> > > >http://code.google.com/p/openmeetings/source/detail?r=3988 >> >> > >> >> > >> >> >>> >>> > Modified: >> >> > >> >> > >> >> >>> >>> > >> >> > >> >> >>> >>> > >> >> > >> >> >>> >>> > /trunk/singlewebapp/WebContent/openmeetings/base/components/upload/fileUpload.lzx >> >> > >> >> > >> >> >>> >>> > ======================================= >> >> > >> >> >>> >>> > --- >> >> > >> >> > >> >> >>> >>> > >> >> > >> >> >>> >>> > >> >> > >> >> >>> >>> > /trunk/singlewebapp/WebContent/openmeetings/base/components/upload/fileUpload.lzx >> >> > >> >> >>> >>> > Fri Jun 24 08:16:56 2011 >> >> > >> >> >>> >>> > +++ >> >> > >> >> > >> >> >>> >>> > >> >> > >> >> >>> >>> > >> >> > >> >> >>> >>> > /trunk/singlewebapp/WebContent/openmeetings/base/components/upload/fileUpload.lzx >> >> > >> >> >>> >>> > Sat Jul 16 02:59:13 2011 >> >> > >> >> >>> >>> > @@ -64,7 +64,7 @@ >> >> > >> >> > >> >> >>> >>> > var imageTypes = new >> >> > >> >> >>> >>> > Object(); >> >> > >> >> >>> >>> > imageTypes.description >> >> > >> >> >>> >>> > = >> >> > >> >> >>> >>> > "Allowed >> >> > >> >> >>> >>> > Files"; >> >> > >> >> >>> >>> > - imageTypes.extension >> >> > >> >> >>> >>> > = >> >> > >> >> >>> >>> > "*.txt; >> >> > >> >> >>> >>> > *.pcd; >> >> > >> >> >>> >>> > *.ps; >> >> > >> >> >>> >>> > *.psd; *.tiff; *.bmp; *.svg; *.jpg; *.jpeg; *.gif; >> >> > >> >> >>> >>> > *.png; *.ppt; >> >> > >> >> >>> >>> > *.pptx; >> >> > >> >> >>> >>> > *.odp; *.odt; *.doc; *.docx; *.rtf; *.txt; *.ods; >> >> > >> >> >>> >>> > *.sxc; >> >> > >> >> >>> >>> > *.xls; >> >> > >> >> >>> >>> > *.xlsx; >> >> > >> >> >>> >>> > *.pdf"; >> >> > >> >> >>> >>> > + imageTypes.extension >> >> > >> >> >>> >>> > = >> >> > >> >> >>> >>> > "*.txt; >> >> > >> >> >>> >>> > *.ppt; >> >> > >> >> >>> >>> > *.pptx; *.odp; *.odt; *.wpd; *.doc; *.docx; *.rtf; >> >> > >> >> >>> >>> > *.txt; *.ods; >> >> > >> >> >>> >>> > *.sxc; >> >> > >> >> >>> >>> > *.xls; *.xlsx; *.sxi; *.pdf; *.avi; *.mov; *.flv; >> >> > >> >> >>> >>> > *.mp4; >> >> > >> >> >>> >>> > *.tga; >> >> > >> >> >>> >>> > *.xcf; >> >> > >> >> >>> >>> > *.wpg; *.ico; *.ttf; *.pcd; *.pcds; *.ps; *.psd; >> >> > >> >> >>> >>> > *.tiff; >> >> > >> >> >>> >>> > *.bmp; >> >> > >> >> >>> >>> > *.svg; >> >> > >> >> >>> >>> > *.dpx; *.exr; *.jpg; *.jpeg; *.gif; *.png"; >> >> > >> >> >>> >>> > >> >> > >> >> >>> >>> > allTypes.push(imageTypes); >> >> > >> >> > >> >> >>> >>> > imageTypes = new >> >> > >> >> >>> >>> > Object(); >> >> > >> >> >>> >>> > @@ -78,7 +78,7 @@ >> >> > >> >> >>> >>> > allTypes.push(imageTypes); >> >> > >> >> > >> >> >>> >>> > imageTypes = new >> >> > >> >> >>> >>> > Object(); >> >> > >> >> >>> >>> > - >> >> > >> >> > ... >> >> > >> >> > read more » >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "OpenMeetings developers" group. >> >> To post to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >> http://groups.google.com/group/openmeetings-dev?hl=en. >> >> >> > >> > >> > >> > -- >> > WBR >> > Maxim aka solomax >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "OpenMeetings developers" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/openmeetings-dev?hl=en. >> > >> >> >> >> -- >> Sebastian Wagner >> http://www.webbase-design.de >> http://openmeetings.googlecode.com >> http://www.wagner-sebastian.com >> [email protected] >> >> -- >> You received this message because you are subscribed to the Google Groups >> "OpenMeetings developers" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/openmeetings-dev?hl=en. >> > > > > -- > WBR > Maxim aka solomax (icq #20779631; skype maxim.solodovnik) > mobile phone: +79133861155 > > -- > You received this message because you are subscribed to the Google Groups > "OpenMeetings developers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/openmeetings-dev?hl=en. > -- Sebastian Wagner http://www.webbase-design.de http://openmeetings.googlecode.com http://www.wagner-sebastian.com [email protected] -- You received this message because you are subscribed to the Google Groups "OpenMeetings developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/openmeetings-dev?hl=en.
