Hello Sebastian, I have added mapping for ID’s for import. Maxim should be submitted my changes today. Now I am working on copy the folder of the upload-directory from the Backup-ZIP to the disk.
Vasiliy. On Jul 21, 4:21 pm, "[email protected]" <[email protected]> wrote: > Hi Maxim, > > yes this was the thing that I tried to avoid :) Collecting all IDs and > building lookup tables to see which old id corresponds to the record > after the import. > > The other thing, to import the objects in the correct order is already > implemented. > > Some Issue will be also the uploaded files: There are some > restrictions on the user's profile images. I think they have the > userId as folderName in the path. So you can't copy the folders > anymore into the disk just by coping the ROOT folder of the > upload-directory from the Backup-ZIP to the disk, cause there are some > subfolders that contain invalid folderNames that need to be > rewritten/renamed. > > We can try that out, its obviously better if it works but its a rather > big change. > My concern is the same like yours: Could be quite memory hungry process. > > I got for example a database with 40.000++ users that I need to > import/export. Holding all 40.000 users would not work, maybe holding > the 40.000 IDs could work. > > Sebastian > > 2011/7/21 Maxim Solodovnik <[email protected]>: > > > > > > > > > Hello Sebastian, Vasiliy > > Instead of creating generator classes I would like to propose the following > > mechanism: > > 1) Import should be done in "well known" order i.e. organisanisations are > > imported first then users, then rooms etc. > > 2) While importing certain object id is not set. Instead of this old id and > > new id are stored in hashtable key== oldId, value == newId > > 3) While importing objects containing links to "parent objects" newId is > > used using Hashtable created. > > 4) Pictures/streams/videos are stored in the file system using newId > > The import will be memory consuming operation but all constraints will be > > valid. > > > On Mon, Jul 18, 2011 at 17:46, Vasya <[email protected]> wrote: > > >> Hi Sebastian, > > >> Thanks for your answer! > > >> I will continue to investigate problem tomorrow. > > >> Vasiliy. > > >> On Jul 18, 5:32 pm, "[email protected]" <[email protected]> > >> wrote: > >> > Hi Vasya, > > >> > If tested getBackupRooms yesterday, it was okay for me. > > >> > The thing with the *deleted* is that it has to return ALL records, no > >> > matter if the deleted flag is set or not. > >> > This is the case because of the foreign key references. > > >> > Just imagine the following sample: > >> > Rooms-Table: > >> > ID name deleted > >> > 1 room1 false > >> > 2 room2 true > >> > 3 room3 false > > >> > Rooms-Organizations Table: > >> > ID rooms_id organisation_id > >> > 1 1 1 > >> > 1 2 2 > >> > 1 3 1 > > >> > If you only export those records that are deleted == 'false' > >> > => the importer would add room3 as the record with the ID 2 and as > >> > consequence the room would be assigned to the organization 2 => which > >> > is wrong of course. > > >> > And the same you have between all tables. That is why I did import > >> > records always WITH all records marked as "deleted" AND in the import > >> > process I did add each object with the ID ALREADY SET. That means I > >> > did not use the generated Id from Hibernate, I did use the ID that was > >> > provided with the XML. But this does not work anymore as openJPA does > >> > not accept objects where the ID is > 0. > >> > At the moment everything is fine if you do a single import directly > >> > after the installation. But for example a user creates an Organization > >> > (will be Id = 2), deletes it again, the ID=2 is gone, if we do any > >> > import now with references to an organization=2 => everything Fails. > >> > In the old implementation we still where able to add an Org with ID=2 > >> > cause we did provide the ID in the Import Object and did not use the > >> > generated one for objects that are created using the import. > > >> > That is the problem, if you have an idea on what we might can do about > >> > those ID problem, maybe there is a Generator Class in openJPA that > >> > accepts self generated IDs. > > >> > Sebastian > > >> > 2011/7/18 Vasya <[email protected]>: > > >> > > Hi Sebastian, > > >> > > I have found and fixed two errors in backup: > >> > > 1. Roommanagement.getInstance().getBackupRooms() returns error so > >> > > root > >> > > condition of criteria query is not defined. > >> > > 2. BackupImport.getAppointmentListByXML method does not work > >> > > properly > >> > > because ‘user_id’ is not set. > >> > > I have fixed these errors and sent you email with patch. > > >> > > So, I have question about > >> > > UsersDaoImpl.getInstance().getAllUsersDeleted() method. It’s not > >> > > cleared for me what list of users should be returns all users or > >> > > deleted users? This method calls from BackupExport for export users. > >> > > Please clarify what list of users should be return from this method. > > >> > > Tomorrow I will continue to work on openmeetings and try to resolve > >> > > other problems. > > >> > > Thanks, > >> > > Vasiliy. > > >> > > On Jul 18, 1:55 pm, "[email protected]" <[email protected]> > >> > > wrote: > >> > >> Hi Vasya, > > >> > >> I think I've fixed it, but some more testing would be good. > >> > >> I also found out that sometimes emails contain the char:  (and > >> > >>  and  ) > >> > >> ... the SaxParser fails to read those files > >> > >> ... I think it is because people copy/paste their email from > >> > >> somewhere > >> > >> in the fields. > >> > >> But I have no idea about how they do it ... However this only happens > >> > >> 1 time per 1000 users but still strange. > > >> > >> For the Primary Key Issue I have no idea too, I changed it now but as > >> > >> you can imagine it can get quite confusing if the system generates > >> > >> the > >> > >> id itself and you import xxx records that "should" reference each > >> > >> other based on those IDs. For example rooms VS organisations or users > >> > >> VS organizations > > >> > >> Sebastian > > >> > >> 2011/7/18 Vasya <[email protected]>: > > >> > >> > Hello Sebastian, > > >> > >> > May be my test of backup was not completely. Today a have try to > >> > >> > retest the import of backup more details. > > >> > >> > Thanks, > > >> > >> > Vasily. > > >> > >> > On Jul 18, 1:12 am, "[email protected]" <[email protected]> > >> > >> > wrote: > >> > >> >> The import of Backup did not work. OpenJPA does not accept to add > >> > >> >> a > >> > >> >> new object where the Primary Key is already set. This is > >> > >> >> neccessary > >> > >> >> for importing existing Records from one database to another > >> > >> >> because of > >> > >> >> the foreign key relation-ships. > > >> > >> >> I have changed the code for now so that it works again. (in r3997) > >> > >> >> However, there are still some issues to be solved. > > >> > >> >> You might never run into that, the imported checks if the Id does > >> > >> >> exists before it adds a new record. So if you export a freshly > >> > >> >> installed openmeetings and then import it again into the same => > >> > >> >> Zero > >> > >> >> happens. > >> > >> >> So to test the backup import you really need to generate some > >> > >> >> users/conferencerooms/calendar-events/private messages and > >> > >> >> contacts et > >> > >> >> cetera and then trying to import that backup into a fresh > >> > >> >> installed > >> > >> >> openmeetings database. > > >> > >> >> Sebastian > > >> > >> >> 2011/7/16 [email protected] <[email protected]>: > > >> > >> >> > the saluation thing is fixed too.... > > >> > >> >> > Sebastian > > >> > >> >> > 2011/7/16 Maxim Solodovnik <[email protected]>: > >> > >> >> >> great to hear it :) > >> > >> >> >> please let me know in case of any issues you will found. > > >> > >> >> >> On Sat, Jul 16, 2011 at 20:27, [email protected] > >> > >> >> >> <[email protected]> > >> > >> >> >> wrote: > > >> > >> >> >>> ok ... > > >> > >> >> >>> I've fixed that issue with the order. Now I need to add also > >> > >> >> >>> the where > >> > >> >> >>> clause in the annotation. This is really nice, this manual > >> > >> >> >>> sub-select > >> > >> >> >>> for the main-menu always was a nasty hack because the x-doclet > >> > >> >> >>> hbm.xml > >> > >> >> >>> creation did not allow those sub-queries with where+orderby. > >> > >> >> >>> ... this > >> > >> >> >>> is also the reason why we have STRING flag *deleted* and NOT a > >> > >> >> >>> boolean > >> > >> >> >>> field *deleted*. In previous hibernate/xdoclet creation it was > >> > >> >> >>> not > >> > >> >> >>> possible to write a annotation with a boolean value as where > >> > >> >> >>> clause ^^ > >> > >> >> >>> At some point we might clean that up too. > > >> > >> >> >>> The other one with the strang saluations_id / Primary Key is > >> > >> >> >>> still to be > >> > >> >> >>> done. > > >> > >> >> >>> Sebastian > > >> > >> >> >>> 2011/7/16 [email protected] <[email protected]>: > >> > >> >> >>> > Hi Maxim, > > >> > >> >> >>> > I am currently checking out the openJPA refactoring. I like > >> > >> >> >>> > it, looks > >> > >> >> >>> > very nice and using annotations instead of XML files is > >> > >> >> >>> > quite elegant. > > >> > >> >> >>> > Maybe you have an idea about those things: > >> > >> >> >>> > - The navigation items are ordered different. For example by > >> > >> >> >>> > default > >> > >> >> >>> > you have the calendar instead of the dashboard now. And in > >> > >> >> >>> > the > >> > >> >> >>> > Administration you have a different order too. You have some > >> > >> >> >>> > code to > >> > >> >> >>> > do the one-to-many relation in your mapping. I think we > >> > >> >> >>> > might remove > >> > >> >> >>> > that and use java.util.List again, I think java.util.Set is > >> > >> >> >>> > not save > >> > >> >> >>> > in meaning of the order of items. > >> > >> >> >>> > - The table saluations start with the id 251 in my case. > >> > >> >> >>> > Normally > >> > >> >> >>> > record 1 should have salutations_id 1. That is necessary as > >> > >> >> >>> > the its > >> > >> >> >>> > hardcoded in the client adminstration panel that "Mr" has > >> > >> >> >>> > the ID 1. > >> > >> >> >>> > You got an idea about that? > > >> > >> >> >>> > Sebastian > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "OpenMeetings developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/openmeetings-dev?hl=en.
