Hello All, Thanks for advice! I will try to add mapping for import. Sebastian, can you give me backup-ZIP with large amount of users for testing.
Thanks, Vasiliy. On Jul 21, 10:45 pm, "[email protected]" <[email protected]> wrote: > Yeah, > > you're right. I would suggest if it works with a 40.000 Users > import/export, at lets say -Xmx512MB for the hole red5 server process, > I think its an acceptable performance. > > Sebastian > > 2011/7/21 Maxim Solodovnik <[email protected]>: > > > > > > > > > If memory used amount of memory will be unacceptably high we can use > > temporary tables for store id pairs. > > With openJPA it will be just couple additional beans. > > > On Thu, Jul 21, 2011 at 16:21, [email protected] <[email protected]> > > wrote: > > >> Hi Maxim, > > >> yes this was the thing that I tried to avoid :) Collecting all IDs and > >> building lookup tables to see which old id corresponds to the record > >> after the import. > > >> The other thing, to import the objects in the correct order is already > >> implemented. > > >> Some Issue will be also the uploaded files: There are some > >> restrictions on the user's profile images. I think they have the > >> userId as folderName in the path. So you can't copy the folders > >> anymore into the disk just by coping the ROOT folder of the > >> upload-directory from the Backup-ZIP to the disk, cause there are some > >> subfolders that contain invalid folderNames that need to be > >> rewritten/renamed. > > >> We can try that out, its obviously better if it works but its a rather > >> big change. > >> My concern is the same like yours: Could be quite memory hungry process. > > >> I got for example a database with 40.000++ users that I need to > >> import/export. Holding all 40.000 users would not work, maybe holding > >> the 40.000 IDs could work. > > >> Sebastian > > >> 2011/7/21 Maxim Solodovnik <[email protected]>: > >> > Hello Sebastian, Vasiliy > >> > Instead of creating generator classes I would like to propose the > >> > following > >> > mechanism: > >> > 1) Import should be done in "well known" order i.e. organisanisations > >> > are > >> > imported first then users, then rooms etc. > >> > 2) While importing certain object id is not set. Instead of this old id > >> > and > >> > new id are stored in hashtable key== oldId, value == newId > >> > 3) While importing objects containing links to "parent objects" newId is > >> > used using Hashtable created. > >> > 4) Pictures/streams/videos are stored in the file system using newId > >> > The import will be memory consuming operation but all constraints will > >> > be > >> > valid. > > >> > On Mon, Jul 18, 2011 at 17:46, Vasya <[email protected]> wrote: > > >> >> Hi Sebastian, > > >> >> Thanks for your answer! > > >> >> I will continue to investigate problem tomorrow. > > >> >> Vasiliy. > > >> >> On Jul 18, 5:32 pm, "[email protected]" <[email protected]> > >> >> wrote: > >> >> > Hi Vasya, > > >> >> > If tested getBackupRooms yesterday, it was okay for me. > > >> >> > The thing with the *deleted* is that it has to return ALL records, no > >> >> > matter if the deleted flag is set or not. > >> >> > This is the case because of the foreign key references. > > >> >> > Just imagine the following sample: > >> >> > Rooms-Table: > >> >> > ID name deleted > >> >> > 1 room1 false > >> >> > 2 room2 true > >> >> > 3 room3 false > > >> >> > Rooms-Organizations Table: > >> >> > ID rooms_id organisation_id > >> >> > 1 1 1 > >> >> > 1 2 2 > >> >> > 1 3 1 > > >> >> > If you only export those records that are deleted == 'false' > >> >> > => the importer would add room3 as the record with the ID 2 and as > >> >> > consequence the room would be assigned to the organization 2 => which > >> >> > is wrong of course. > > >> >> > And the same you have between all tables. That is why I did import > >> >> > records always WITH all records marked as "deleted" AND in the import > >> >> > process I did add each object with the ID ALREADY SET. That means I > >> >> > did not use the generated Id from Hibernate, I did use the ID that > >> >> > was > >> >> > provided with the XML. But this does not work anymore as openJPA does > >> >> > not accept objects where the ID is > 0. > >> >> > At the moment everything is fine if you do a single import directly > >> >> > after the installation. But for example a user creates an > >> >> > Organization > >> >> > (will be Id = 2), deletes it again, the ID=2 is gone, if we do any > >> >> > import now with references to an organization=2 => everything Fails. > >> >> > In the old implementation we still where able to add an Org with ID=2 > >> >> > cause we did provide the ID in the Import Object and did not use the > >> >> > generated one for objects that are created using the import. > > >> >> > That is the problem, if you have an idea on what we might can do > >> >> > about > >> >> > those ID problem, maybe there is a Generator Class in openJPA that > >> >> > accepts self generated IDs. > > >> >> > Sebastian > > >> >> > 2011/7/18 Vasya <[email protected]>: > > >> >> > > Hi Sebastian, > > >> >> > > I have found and fixed two errors in backup: > >> >> > > 1. Roommanagement.getInstance().getBackupRooms() returns error > >> >> > > so > >> >> > > root > >> >> > > condition of criteria query is not defined. > >> >> > > 2. BackupImport.getAppointmentListByXML method does not work > >> >> > > properly > >> >> > > because ‘user_id’ is not set. > >> >> > > I have fixed these errors and sent you email with patch. > > >> >> > > So, I have question about > >> >> > > UsersDaoImpl.getInstance().getAllUsersDeleted() method. It’s not > >> >> > > cleared for me what list of users should be returns all users or > >> >> > > deleted users? This method calls from BackupExport for export > >> >> > > users. > >> >> > > Please clarify what list of users should be return from this > >> >> > > method. > > >> >> > > Tomorrow I will continue to work on openmeetings and try to resolve > >> >> > > other problems. > > >> >> > > Thanks, > >> >> > > Vasiliy. > > >> >> > > On Jul 18, 1:55 pm, "[email protected]" <[email protected]> > >> >> > > wrote: > >> >> > >> Hi Vasya, > > >> >> > >> I think I've fixed it, but some more testing would be good. > >> >> > >> I also found out that sometimes emails contain the char:  (and > >> >> > >>  and  ) > >> >> > >> ... the SaxParser fails to read those files > >> >> > >> ... I think it is because people copy/paste their email from > >> >> > >> somewhere > >> >> > >> in the fields. > >> >> > >> But I have no idea about how they do it ... However this only > >> >> > >> happens > >> >> > >> 1 time per 1000 users but still strange. > > >> >> > >> For the Primary Key Issue I have no idea too, I changed it now but > >> >> > >> as > >> >> > >> you can imagine it can get quite confusing if the system generates > >> >> > >> the > >> >> > >> id itself and you import xxx records that "should" reference each > >> >> > >> other based on those IDs. For example rooms VS organisations or > >> >> > >> users > >> >> > >> VS organizations > > >> >> > >> Sebastian > > >> >> > >> 2011/7/18 Vasya <[email protected]>: > > >> >> > >> > Hello Sebastian, > > >> >> > >> > May be my test of backup was not completely. Today a have try to > >> >> > >> > retest the import of backup more details. > > >> >> > >> > Thanks, > > >> >> > >> > Vasily. > > >> >> > >> > On Jul 18, 1:12 am, "[email protected]" > >> >> > >> > <[email protected]> > >> >> > >> > wrote: > >> >> > >> >> The import of Backup did not work. OpenJPA does not accept to > >> >> > >> >> add > >> >> > >> >> a > >> >> > >> >> new object where the Primary Key is already set. This is > >> >> > >> >> neccessary > >> >> > >> >> for importing existing Records from one database to another > >> >> > >> >> because of > >> >> > >> >> the foreign key relation-ships. > > >> >> > >> >> I have changed the code for now so that it works again. (in > >> >> > >> >> r3997) > >> >> > >> >> However, there are still some issues to be solved. > > >> >> > >> >> You might never run into that, the imported checks if the Id > >> >> > >> >> does > >> >> > >> >> exists before it adds a new record. So if you export a freshly > >> >> > >> >> installed openmeetings and then import it again into the same > >> >> > >> >> => > >> >> > >> >> Zero > >> >> > >> >> happens. > >> >> > >> >> So to test the backup import you really need to generate some > >> >> > >> >> users/conferencerooms/calendar-events/private messages and > >> >> > >> >> contacts et > >> >> > >> >> cetera and then trying to import that backup into a fresh > >> >> > >> >> installed > >> >> > >> >> openmeetings database. > > >> >> > >> >> Sebastian > > >> >> > >> >> 2011/7/16 [email protected] <[email protected]>: > > >> >> > >> >> > the saluation thing is fixed too.... > > >> >> > >> >> > Sebastian > > >> >> > >> >> > 2011/7/16 Maxim Solodovnik <[email protected]>: > >> >> > >> >> >> great to hear it :) > >> >> > >> >> >> please let me know in case of any issues you will found. > > >> >> > >> >> >> On Sat, Jul 16, 2011 at 20:27, [email protected] > >> >> > >> >> >> <[email protected]> > >> >> > >> >> >> wrote: > > >> >> > >> >> >>> ok ... > > >> >> > >> >> >>> I've fixed that issue with the order. Now I need to add > >> >> > >> >> >>> also > >> >> > >> >> >>> the where > >> >> > >> >> >>> clause in the annotation. This is really nice, this manual > >> >> > >> >> >>> sub-select > >> >> > >> >> >>> for the main-menu always was a nasty hack because the > >> >> > >> >> >>> x-doclet > >> >> > >> >> >>> hbm.xml > >> >> > >> >> >>> creation did not allow those sub-queries with > >> >> > >> >> >>> where+orderby. > >> >> > >> >> >>> ... this > >> >> > >> >> >>> is also the reason why we have STRING flag *deleted* and > >> >> > >> >> >>> NOT a > >> >> > >> >> >>> boolean > >> >> > >> >> >>> field *deleted*. In previous hibernate/xdoclet creation it > >> >> > >> >> >>> was > >> >> > >> >> >>> not > >> >> > >> >> >>> possible to write a annotation with a boolean value as > >> >> > >> >> >>> where > >> >> > >> >> >>> clause ^^ > >> >> > >> >> >>> At some point we might clean that up too. > > >> >> > >> >> >>> The other > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "OpenMeetings developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/openmeetings-dev?hl=en.
