Okay Vasya, regarding the ZIP for the 40.000 Users....this import is not the usual test-case
I do not export the Files in it, cause that would result in a 80GB ZIP file. I do only import/export the database-record in the ZIP. I have added some option for that. You should generate some sample data with some sample rooms. >From what I understood your import would allow users to import the same ZIP n-times. Or do you add some checks to not duplicate it? What happens to users if you import a user twice, cause the username/login would exist two times and the Auth-Mechanism would not work anymore. I think for a phase1 its okay if that fails, somebody should know that importing the same ZIP twice can result in duplicates ... Sebastian 2011/7/22 Vasya <[email protected]>: > Hello Sebastian, > > I have added mapping for ID’s for import. Maxim should be submitted my > changes today. > Now I am working on copy the folder of the upload-directory from the > Backup-ZIP to the disk. > > Vasiliy. > > > On Jul 21, 4:21 pm, "[email protected]" <[email protected]> > wrote: >> Hi Maxim, >> >> yes this was the thing that I tried to avoid :) Collecting all IDs and >> building lookup tables to see which old id corresponds to the record >> after the import. >> >> The other thing, to import the objects in the correct order is already >> implemented. >> >> Some Issue will be also the uploaded files: There are some >> restrictions on the user's profile images. I think they have the >> userId as folderName in the path. So you can't copy the folders >> anymore into the disk just by coping the ROOT folder of the >> upload-directory from the Backup-ZIP to the disk, cause there are some >> subfolders that contain invalid folderNames that need to be >> rewritten/renamed. >> >> We can try that out, its obviously better if it works but its a rather >> big change. >> My concern is the same like yours: Could be quite memory hungry process. >> >> I got for example a database with 40.000++ users that I need to >> import/export. Holding all 40.000 users would not work, maybe holding >> the 40.000 IDs could work. >> >> Sebastian >> >> 2011/7/21 Maxim Solodovnik <[email protected]>: >> >> >> >> >> >> >> >> > Hello Sebastian, Vasiliy >> > Instead of creating generator classes I would like to propose the following >> > mechanism: >> > 1) Import should be done in "well known" order i.e. organisanisations are >> > imported first then users, then rooms etc. >> > 2) While importing certain object id is not set. Instead of this old id and >> > new id are stored in hashtable key== oldId, value == newId >> > 3) While importing objects containing links to "parent objects" newId is >> > used using Hashtable created. >> > 4) Pictures/streams/videos are stored in the file system using newId >> > The import will be memory consuming operation but all constraints will be >> > valid. >> >> > On Mon, Jul 18, 2011 at 17:46, Vasya <[email protected]> wrote: >> >> >> Hi Sebastian, >> >> >> Thanks for your answer! >> >> >> I will continue to investigate problem tomorrow. >> >> >> Vasiliy. >> >> >> On Jul 18, 5:32 pm, "[email protected]" <[email protected]> >> >> wrote: >> >> > Hi Vasya, >> >> >> > If tested getBackupRooms yesterday, it was okay for me. >> >> >> > The thing with the *deleted* is that it has to return ALL records, no >> >> > matter if the deleted flag is set or not. >> >> > This is the case because of the foreign key references. >> >> >> > Just imagine the following sample: >> >> > Rooms-Table: >> >> > ID name deleted >> >> > 1 room1 false >> >> > 2 room2 true >> >> > 3 room3 false >> >> >> > Rooms-Organizations Table: >> >> > ID rooms_id organisation_id >> >> > 1 1 1 >> >> > 1 2 2 >> >> > 1 3 1 >> >> >> > If you only export those records that are deleted == 'false' >> >> > => the importer would add room3 as the record with the ID 2 and as >> >> > consequence the room would be assigned to the organization 2 => which >> >> > is wrong of course. >> >> >> > And the same you have between all tables. That is why I did import >> >> > records always WITH all records marked as "deleted" AND in the import >> >> > process I did add each object with the ID ALREADY SET. That means I >> >> > did not use the generated Id from Hibernate, I did use the ID that was >> >> > provided with the XML. But this does not work anymore as openJPA does >> >> > not accept objects where the ID is > 0. >> >> > At the moment everything is fine if you do a single import directly >> >> > after the installation. But for example a user creates an Organization >> >> > (will be Id = 2), deletes it again, the ID=2 is gone, if we do any >> >> > import now with references to an organization=2 => everything Fails. >> >> > In the old implementation we still where able to add an Org with ID=2 >> >> > cause we did provide the ID in the Import Object and did not use the >> >> > generated one for objects that are created using the import. >> >> >> > That is the problem, if you have an idea on what we might can do about >> >> > those ID problem, maybe there is a Generator Class in openJPA that >> >> > accepts self generated IDs. >> >> >> > Sebastian >> >> >> > 2011/7/18 Vasya <[email protected]>: >> >> >> > > Hi Sebastian, >> >> >> > > I have found and fixed two errors in backup: >> >> > > 1. Roommanagement.getInstance().getBackupRooms() returns error so >> >> > > root >> >> > > condition of criteria query is not defined. >> >> > > 2. BackupImport.getAppointmentListByXML method does not work >> >> > > properly >> >> > > because ‘user_id’ is not set. >> >> > > I have fixed these errors and sent you email with patch. >> >> >> > > So, I have question about >> >> > > UsersDaoImpl.getInstance().getAllUsersDeleted() method. It’s not >> >> > > cleared for me what list of users should be returns all users or >> >> > > deleted users? This method calls from BackupExport for export users. >> >> > > Please clarify what list of users should be return from this method. >> >> >> > > Tomorrow I will continue to work on openmeetings and try to resolve >> >> > > other problems. >> >> >> > > Thanks, >> >> > > Vasiliy. >> >> >> > > On Jul 18, 1:55 pm, "[email protected]" <[email protected]> >> >> > > wrote: >> >> > >> Hi Vasya, >> >> >> > >> I think I've fixed it, but some more testing would be good. >> >> > >> I also found out that sometimes emails contain the char:  (and >> >> > >>  and  ) >> >> > >> ... the SaxParser fails to read those files >> >> > >> ... I think it is because people copy/paste their email from >> >> > >> somewhere >> >> > >> in the fields. >> >> > >> But I have no idea about how they do it ... However this only happens >> >> > >> 1 time per 1000 users but still strange. >> >> >> > >> For the Primary Key Issue I have no idea too, I changed it now but as >> >> > >> you can imagine it can get quite confusing if the system generates >> >> > >> the >> >> > >> id itself and you import xxx records that "should" reference each >> >> > >> other based on those IDs. For example rooms VS organisations or users >> >> > >> VS organizations >> >> >> > >> Sebastian >> >> >> > >> 2011/7/18 Vasya <[email protected]>: >> >> >> > >> > Hello Sebastian, >> >> >> > >> > May be my test of backup was not completely. Today a have try to >> >> > >> > retest the import of backup more details. >> >> >> > >> > Thanks, >> >> >> > >> > Vasily. >> >> >> > >> > On Jul 18, 1:12 am, "[email protected]" <[email protected]> >> >> > >> > wrote: >> >> > >> >> The import of Backup did not work. OpenJPA does not accept to add >> >> > >> >> a >> >> > >> >> new object where the Primary Key is already set. This is >> >> > >> >> neccessary >> >> > >> >> for importing existing Records from one database to another >> >> > >> >> because of >> >> > >> >> the foreign key relation-ships. >> >> >> > >> >> I have changed the code for now so that it works again. (in r3997) >> >> > >> >> However, there are still some issues to be solved. >> >> >> > >> >> You might never run into that, the imported checks if the Id does >> >> > >> >> exists before it adds a new record. So if you export a freshly >> >> > >> >> installed openmeetings and then import it again into the same => >> >> > >> >> Zero >> >> > >> >> happens. >> >> > >> >> So to test the backup import you really need to generate some >> >> > >> >> users/conferencerooms/calendar-events/private messages and >> >> > >> >> contacts et >> >> > >> >> cetera and then trying to import that backup into a fresh >> >> > >> >> installed >> >> > >> >> openmeetings database. >> >> >> > >> >> Sebastian >> >> >> > >> >> 2011/7/16 [email protected] <[email protected]>: >> >> >> > >> >> > the saluation thing is fixed too.... >> >> >> > >> >> > Sebastian >> >> >> > >> >> > 2011/7/16 Maxim Solodovnik <[email protected]>: >> >> > >> >> >> great to hear it :) >> >> > >> >> >> please let me know in case of any issues you will found. >> >> >> > >> >> >> On Sat, Jul 16, 2011 at 20:27, [email protected] >> >> > >> >> >> <[email protected]> >> >> > >> >> >> wrote: >> >> >> > >> >> >>> ok ... >> >> >> > >> >> >>> I've fixed that issue with the order. Now I need to add also >> >> > >> >> >>> the where >> >> > >> >> >>> clause in the annotation. This is really nice, this manual >> >> > >> >> >>> sub-select >> >> > >> >> >>> for the main-menu always was a nasty hack because the x-doclet >> >> > >> >> >>> hbm.xml >> >> > >> >> >>> creation did not allow those sub-queries with where+orderby. >> >> > >> >> >>> ... this >> >> > >> >> >>> is also the reason why we have STRING flag *deleted* and NOT a >> >> > >> >> >>> boolean >> >> > >> >> >>> field *deleted*. In previous hibernate/xdoclet creation it was >> >> > >> >> >>> not >> >> > >> >> >>> possible to write a annotation with a boolean value as where >> >> > >> >> >>> clause ^^ >> >> > >> >> >>> At some point we might clean that up too. >> >> >> > >> >> >>> The other one with the strang saluations_id / Primary Key is >> >> > >> >> >>> still to be >> >> > >> >> >>> done. >> >> >> > >> >> >>> Sebastian >> >> >> > >> >> >>> 2011/7/16 [email protected] <[email protected]>: >> >> > >> >> >>> > Hi Maxim, >> >> >> > >> >> >>> > I am currently checking out the openJPA refactoring. I like >> >> > >> >> >>> > it, looks >> >> > >> >> >>> > very nice and using annotations instead of XML files is >> >> > >> >> >>> > quite elegant. >> >> >> > >> >> >>> > Maybe you have an idea about those things: >> >> > >> >> >>> > - The navigation items are ordered different. For example by >> >> > >> >> >>> > default >> >> > >> >> >>> > you have the calendar instead of the dashboard now. And in >> >> > >> >> >>> > the >> >> > >> >> >>> > Administration you have a different order too. You have some >> >> > >> >> >>> > code to >> >> > >> >> >>> > do the one-to-many relation in your mapping. I think we >> >> > >> >> >>> > might remove >> >> > >> >> >>> > that and use java.util.List again, I think java.util.Set is >> >> > >> >> >>> > not save >> >> > >> >> >>> > in meaning of the order of items. >> >> > >> >> >>> > - The table saluations start with the id 251 in my case. >> >> > >> >> >>> > Normally >> >> > >> >> >>> > record 1 should have salutations_id 1. That is necessary as >> >> > >> >> >>> > the its >> >> > >> >> >>> > hardcoded in the client adminstration panel that "Mr" has >> >> > >> >> >>> > the ID 1. >> >> > >> >> >>> > You got an idea about that? >> >> >> > >> >> >>> > Sebastian >> >> ... >> >> read more » > > -- > You received this message because you are subscribed to the Google Groups > "OpenMeetings developers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/openmeetings-dev?hl=en. > > -- Sebastian Wagner http://www.webbase-design.de http://openmeetings.googlecode.com http://www.wagner-sebastian.com [email protected] -- You received this message because you are subscribed to the Google Groups "OpenMeetings developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/openmeetings-dev?hl=en.
