Hello All, I have checked in the code with id maps, discussed earlier. It has no check for duplicate users (import done multiple times)
I'm not sure how can we protect the DB from such user action .... Even if all data would be checked it would not work in following situation: 1) import was performed 2) some data was edited (using admin) 3) import was run one more time Maybe you have any ideas about it? On Fri, Jul 22, 2011 at 17:42, [email protected] <[email protected]>wrote: > Okay Vasya, > > regarding the ZIP for the 40.000 Users....this import is not the usual > test-case > > I do not export the Files in it, cause that would result in a 80GB ZIP > file. I do only import/export the database-record in the ZIP. I have > added some option for that. > You should generate some sample data with some sample rooms. > > From what I understood your import would allow users to import the > same ZIP n-times. > Or do you add some checks to not duplicate it? > > What happens to users if you import a user twice, cause the > username/login would exist two times and the Auth-Mechanism would not > work anymore. I think for a phase1 its okay if that fails, somebody > should know that importing the same ZIP twice can result in duplicates > ... > > Sebastian > > 2011/7/22 Vasya <[email protected]>: > > Hello Sebastian, > > > > I have added mapping for ID’s for import. Maxim should be submitted my > > changes today. > > Now I am working on copy the folder of the upload-directory from the > > Backup-ZIP to the disk. > > > > Vasiliy. > > > > > > On Jul 21, 4:21 pm, "[email protected]" <[email protected]> > > wrote: > >> Hi Maxim, > >> > >> yes this was the thing that I tried to avoid :) Collecting all IDs and > >> building lookup tables to see which old id corresponds to the record > >> after the import. > >> > >> The other thing, to import the objects in the correct order is already > >> implemented. > >> > >> Some Issue will be also the uploaded files: There are some > >> restrictions on the user's profile images. I think they have the > >> userId as folderName in the path. So you can't copy the folders > >> anymore into the disk just by coping the ROOT folder of the > >> upload-directory from the Backup-ZIP to the disk, cause there are some > >> subfolders that contain invalid folderNames that need to be > >> rewritten/renamed. > >> > >> We can try that out, its obviously better if it works but its a rather > >> big change. > >> My concern is the same like yours: Could be quite memory hungry process. > >> > >> I got for example a database with 40.000++ users that I need to > >> import/export. Holding all 40.000 users would not work, maybe holding > >> the 40.000 IDs could work. > >> > >> Sebastian > >> > >> 2011/7/21 Maxim Solodovnik <[email protected]>: > >> > >> > >> > >> > >> > >> > >> > >> > Hello Sebastian, Vasiliy > >> > Instead of creating generator classes I would like to propose the > following > >> > mechanism: > >> > 1) Import should be done in "well known" order i.e. organisanisations > are > >> > imported first then users, then rooms etc. > >> > 2) While importing certain object id is not set. Instead of this old > id and > >> > new id are stored in hashtable key== oldId, value == newId > >> > 3) While importing objects containing links to "parent objects" newId > is > >> > used using Hashtable created. > >> > 4) Pictures/streams/videos are stored in the file system using newId > >> > The import will be memory consuming operation but all constraints will > be > >> > valid. > >> > >> > On Mon, Jul 18, 2011 at 17:46, Vasya <[email protected]> > wrote: > >> > >> >> Hi Sebastian, > >> > >> >> Thanks for your answer! > >> > >> >> I will continue to investigate problem tomorrow. > >> > >> >> Vasiliy. > >> > >> >> On Jul 18, 5:32 pm, "[email protected]" <[email protected]> > >> >> wrote: > >> >> > Hi Vasya, > >> > >> >> > If tested getBackupRooms yesterday, it was okay for me. > >> > >> >> > The thing with the *deleted* is that it has to return ALL records, > no > >> >> > matter if the deleted flag is set or not. > >> >> > This is the case because of the foreign key references. > >> > >> >> > Just imagine the following sample: > >> >> > Rooms-Table: > >> >> > ID name deleted > >> >> > 1 room1 false > >> >> > 2 room2 true > >> >> > 3 room3 false > >> > >> >> > Rooms-Organizations Table: > >> >> > ID rooms_id organisation_id > >> >> > 1 1 1 > >> >> > 1 2 2 > >> >> > 1 3 1 > >> > >> >> > If you only export those records that are deleted == 'false' > >> >> > => the importer would add room3 as the record with the ID 2 and as > >> >> > consequence the room would be assigned to the organization 2 => > which > >> >> > is wrong of course. > >> > >> >> > And the same you have between all tables. That is why I did import > >> >> > records always WITH all records marked as "deleted" AND in the > import > >> >> > process I did add each object with the ID ALREADY SET. That means I > >> >> > did not use the generated Id from Hibernate, I did use the ID that > was > >> >> > provided with the XML. But this does not work anymore as openJPA > does > >> >> > not accept objects where the ID is > 0. > >> >> > At the moment everything is fine if you do a single import directly > >> >> > after the installation. But for example a user creates an > Organization > >> >> > (will be Id = 2), deletes it again, the ID=2 is gone, if we do any > >> >> > import now with references to an organization=2 => everything > Fails. > >> >> > In the old implementation we still where able to add an Org with > ID=2 > >> >> > cause we did provide the ID in the Import Object and did not use > the > >> >> > generated one for objects that are created using the import. > >> > >> >> > That is the problem, if you have an idea on what we might can do > about > >> >> > those ID problem, maybe there is a Generator Class in openJPA that > >> >> > accepts self generated IDs. > >> > >> >> > Sebastian > >> > >> >> > 2011/7/18 Vasya <[email protected]>: > >> > >> >> > > Hi Sebastian, > >> > >> >> > > I have found and fixed two errors in backup: > >> >> > > 1. Roommanagement.getInstance().getBackupRooms() returns > error so > >> >> > > root > >> >> > > condition of criteria query is not defined. > >> >> > > 2. BackupImport.getAppointmentListByXML method does not work > >> >> > > properly > >> >> > > because ‘user_id’ is not set. > >> >> > > I have fixed these errors and sent you email with patch. > >> > >> >> > > So, I have question about > >> >> > > UsersDaoImpl.getInstance().getAllUsersDeleted() method. It’s not > >> >> > > cleared for me what list of users should be returns all users or > >> >> > > deleted users? This method calls from BackupExport for export > users. > >> >> > > Please clarify what list of users should be return from this > method. > >> > >> >> > > Tomorrow I will continue to work on openmeetings and try to > resolve > >> >> > > other problems. > >> > >> >> > > Thanks, > >> >> > > Vasiliy. > >> > >> >> > > On Jul 18, 1:55 pm, "[email protected]" < > [email protected]> > >> >> > > wrote: > >> >> > >> Hi Vasya, > >> > >> >> > >> I think I've fixed it, but some more testing would be good. > >> >> > >> I also found out that sometimes emails contain the char:  > (and > >> >> > >>  and  ) > >> >> > >> ... the SaxParser fails to read those files > >> >> > >> ... I think it is because people copy/paste their email from > >> >> > >> somewhere > >> >> > >> in the fields. > >> >> > >> But I have no idea about how they do it ... However this only > happens > >> >> > >> 1 time per 1000 users but still strange. > >> > >> >> > >> For the Primary Key Issue I have no idea too, I changed it now > but as > >> >> > >> you can imagine it can get quite confusing if the system > generates > >> >> > >> the > >> >> > >> id itself and you import xxx records that "should" reference > each > >> >> > >> other based on those IDs. For example rooms VS organisations or > users > >> >> > >> VS organizations > >> > >> >> > >> Sebastian > >> > >> >> > >> 2011/7/18 Vasya <[email protected]>: > >> > >> >> > >> > Hello Sebastian, > >> > >> >> > >> > May be my test of backup was not completely. Today a have try > to > >> >> > >> > retest the import of backup more details. > >> > >> >> > >> > Thanks, > >> > >> >> > >> > Vasily. > >> > >> >> > >> > On Jul 18, 1:12 am, "[email protected]" < > [email protected]> > >> >> > >> > wrote: > >> >> > >> >> The import of Backup did not work. OpenJPA does not accept to > add > >> >> > >> >> a > >> >> > >> >> new object where the Primary Key is already set. This is > >> >> > >> >> neccessary > >> >> > >> >> for importing existing Records from one database to another > >> >> > >> >> because of > >> >> > >> >> the foreign key relation-ships. > >> > >> >> > >> >> I have changed the code for now so that it works again. (in > r3997) > >> >> > >> >> However, there are still some issues to be solved. > >> > >> >> > >> >> You might never run into that, the imported checks if the Id > does > >> >> > >> >> exists before it adds a new record. So if you export a > freshly > >> >> > >> >> installed openmeetings and then import it again into the same > => > >> >> > >> >> Zero > >> >> > >> >> happens. > >> >> > >> >> So to test the backup import you really need to generate some > >> >> > >> >> users/conferencerooms/calendar-events/private messages and > >> >> > >> >> contacts et > >> >> > >> >> cetera and then trying to import that backup into a fresh > >> >> > >> >> installed > >> >> > >> >> openmeetings database. > >> > >> >> > >> >> Sebastian > >> > >> >> > >> >> 2011/7/16 [email protected] <[email protected]>: > >> > >> >> > >> >> > the saluation thing is fixed too.... > >> > >> >> > >> >> > Sebastian > >> > >> >> > >> >> > 2011/7/16 Maxim Solodovnik <[email protected]>: > >> >> > >> >> >> great to hear it :) > >> >> > >> >> >> please let me know in case of any issues you will found. > >> > >> >> > >> >> >> On Sat, Jul 16, 2011 at 20:27, [email protected] > >> >> > >> >> >> <[email protected]> > >> >> > >> >> >> wrote: > >> > >> >> > >> >> >>> ok ... > >> > >> >> > >> >> >>> I've fixed that issue with the order. Now I need to add > also > >> >> > >> >> >>> the where > >> >> > >> >> >>> clause in the annotation. This is really nice, this > manual > >> >> > >> >> >>> sub-select > >> >> > >> >> >>> for the main-menu always was a nasty hack because the > x-doclet > >> >> > >> >> >>> hbm.xml > >> >> > >> >> >>> creation did not allow those sub-queries with > where+orderby. > >> >> > >> >> >>> ... this > >> >> > >> >> >>> is also the reason why we have STRING flag *deleted* and > NOT a > >> >> > >> >> >>> boolean > >> >> > >> >> >>> field *deleted*. In previous hibernate/xdoclet creation > it was > >> >> > >> >> >>> not > >> >> > >> >> >>> possible to write a annotation with a boolean value as > where > >> >> > >> >> >>> clause ^^ > >> >> > >> >> >>> At some point we might clean that up too. > >> > >> >> > >> >> >>> The other one with the strang saluations_id / Primary Key > is > >> >> > >> >> >>> still to be > >> >> > >> >> >>> done. > >> > >> >> > >> >> >>> Sebastian > >> > >> >> > >> >> >>> 2011/7/16 [email protected] <[email protected]>: > >> >> > >> >> >>> > Hi Maxim, > >> > >> >> > >> >> >>> > I am currently checking out the openJPA refactoring. I > like > >> >> > >> >> >>> > it, looks > >> >> > >> >> >>> > very nice and using annotations instead of XML files is > >> >> > >> >> >>> > quite elegant. > >> > >> >> > >> >> >>> > Maybe you have an idea about those things: > >> >> > >> >> >>> > - The navigation items are ordered different. For > example by > >> >> > >> >> >>> > default > >> >> > >> >> >>> > you have the calendar instead of the dashboard now. And > in > >> >> > >> >> >>> > the > >> >> > >> >> >>> > Administration you have a different order too. You have > some > >> >> > >> >> >>> > code to > >> >> > >> >> >>> > do the one-to-many relation in your mapping. I think we > >> >> > >> >> >>> > might remove > >> >> > >> >> >>> > that and use java.util.List again, I think > java.util.Set is > >> >> > >> >> >>> > not save > >> >> > >> >> >>> > in meaning of the order of items. > >> >> > >> >> >>> > - The table saluations start with the id 251 in my > case. > >> >> > >> >> >>> > Normally > >> >> > >> >> >>> > record 1 should have salutations_id 1. That is > necessary as > >> >> > >> >> >>> > the its > >> >> > >> >> >>> > hardcoded in the client adminstration panel that "Mr" > has > >> >> > >> >> >>> > the ID 1. > >> >> > >> >> >>> > You got an idea about that? > >> > >> >> > >> >> >>> > Sebastian > >> > >> ... > >> > >> read more » > > > > -- > > You received this message because you are subscribed to the Google Groups > "OpenMeetings developers" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > [email protected]. > > For more options, visit this group at > http://groups.google.com/group/openmeetings-dev?hl=en. > > > > > > > > -- > Sebastian Wagner > http://www.webbase-design.de > http://openmeetings.googlecode.com > http://www.wagner-sebastian.com > [email protected] > > -- > You received this message because you are subscribed to the Google Groups > "OpenMeetings developers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/openmeetings-dev?hl=en. > > -- WBR Maxim aka solomax -- You received this message because you are subscribed to the Google Groups "OpenMeetings developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/openmeetings-dev?hl=en.
