I have done some tests during the weekend. I have some minor changes to the import, the memory consumption was okay, with Xmx512MB the was no Heap-Space exception. I will commit my changes tonight but it should be no problem to merge them if you have updates too, you can proceed in commiting them. The folder renaming, I had no chance to look inside that yet.
.. as well as the activity window. You can also see the renewed proposal for Apache incubation here: http://mail-archives.apache.org/mod_mbox/incubator-general/201107.mbox/browser Sebastian 2011/7/22 [email protected] <[email protected]>: > The only problem I see are the login/email cause: > Login should be unique so that users can login. > We just simply might add a check on that value and skip that user if > it already exists. > Cause otherwise you always have at minimum one duplicate: > The user with the ID=1, cause even if you import it directly after you > did the installation, ONE user always is there. And it would be bad to > duplicate that one, cause you would not be able to login with this > (Admin) user again. > > Sebastian > > 2011/7/22 Maxim Solodovnik <[email protected]>: >> Hello All, >> I have checked in the code with id maps, discussed earlier. >> It has no check for duplicate users (import done multiple times) >> I'm not sure how can we protect the DB from such user action .... >> Even if all data would be checked it would not work in following situation: >> 1) import was performed >> 2) some data was edited (using admin) >> 3) import was run one more time >> Maybe you have any ideas about it? >> On Fri, Jul 22, 2011 at 17:42, [email protected] <[email protected]> >> wrote: >>> >>> Okay Vasya, >>> >>> regarding the ZIP for the 40.000 Users....this import is not the usual >>> test-case >>> >>> I do not export the Files in it, cause that would result in a 80GB ZIP >>> file. I do only import/export the database-record in the ZIP. I have >>> added some option for that. >>> You should generate some sample data with some sample rooms. >>> >>> From what I understood your import would allow users to import the >>> same ZIP n-times. >>> Or do you add some checks to not duplicate it? >>> >>> What happens to users if you import a user twice, cause the >>> username/login would exist two times and the Auth-Mechanism would not >>> work anymore. I think for a phase1 its okay if that fails, somebody >>> should know that importing the same ZIP twice can result in duplicates >>> ... >>> >>> Sebastian >>> >>> 2011/7/22 Vasya <[email protected]>: >>> > Hello Sebastian, >>> > >>> > I have added mapping for ID’s for import. Maxim should be submitted my >>> > changes today. >>> > Now I am working on copy the folder of the upload-directory from the >>> > Backup-ZIP to the disk. >>> > >>> > Vasiliy. >>> > >>> > >>> > On Jul 21, 4:21 pm, "[email protected]" <[email protected]> >>> > wrote: >>> >> Hi Maxim, >>> >> >>> >> yes this was the thing that I tried to avoid :) Collecting all IDs and >>> >> building lookup tables to see which old id corresponds to the record >>> >> after the import. >>> >> >>> >> The other thing, to import the objects in the correct order is already >>> >> implemented. >>> >> >>> >> Some Issue will be also the uploaded files: There are some >>> >> restrictions on the user's profile images. I think they have the >>> >> userId as folderName in the path. So you can't copy the folders >>> >> anymore into the disk just by coping the ROOT folder of the >>> >> upload-directory from the Backup-ZIP to the disk, cause there are some >>> >> subfolders that contain invalid folderNames that need to be >>> >> rewritten/renamed. >>> >> >>> >> We can try that out, its obviously better if it works but its a rather >>> >> big change. >>> >> My concern is the same like yours: Could be quite memory hungry >>> >> process. >>> >> >>> >> I got for example a database with 40.000++ users that I need to >>> >> import/export. Holding all 40.000 users would not work, maybe holding >>> >> the 40.000 IDs could work. >>> >> >>> >> Sebastian >>> >> >>> >> 2011/7/21 Maxim Solodovnik <[email protected]>: >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> > Hello Sebastian, Vasiliy >>> >> > Instead of creating generator classes I would like to propose the >>> >> > following >>> >> > mechanism: >>> >> > 1) Import should be done in "well known" order i.e. organisanisations >>> >> > are >>> >> > imported first then users, then rooms etc. >>> >> > 2) While importing certain object id is not set. Instead of this old >>> >> > id and >>> >> > new id are stored in hashtable key== oldId, value == newId >>> >> > 3) While importing objects containing links to "parent objects" newId >>> >> > is >>> >> > used using Hashtable created. >>> >> > 4) Pictures/streams/videos are stored in the file system using newId >>> >> > The import will be memory consuming operation but all constraints >>> >> > will be >>> >> > valid. >>> >> >>> >> > On Mon, Jul 18, 2011 at 17:46, Vasya <[email protected]> >>> >> > wrote: >>> >> >>> >> >> Hi Sebastian, >>> >> >>> >> >> Thanks for your answer! >>> >> >>> >> >> I will continue to investigate problem tomorrow. >>> >> >>> >> >> Vasiliy. >>> >> >>> >> >> On Jul 18, 5:32 pm, "[email protected]" <[email protected]> >>> >> >> wrote: >>> >> >> > Hi Vasya, >>> >> >>> >> >> > If tested getBackupRooms yesterday, it was okay for me. >>> >> >>> >> >> > The thing with the *deleted* is that it has to return ALL records, >>> >> >> > no >>> >> >> > matter if the deleted flag is set or not. >>> >> >> > This is the case because of the foreign key references. >>> >> >>> >> >> > Just imagine the following sample: >>> >> >> > Rooms-Table: >>> >> >> > ID name deleted >>> >> >> > 1 room1 false >>> >> >> > 2 room2 true >>> >> >> > 3 room3 false >>> >> >>> >> >> > Rooms-Organizations Table: >>> >> >> > ID rooms_id organisation_id >>> >> >> > 1 1 1 >>> >> >> > 1 2 2 >>> >> >> > 1 3 1 >>> >> >>> >> >> > If you only export those records that are deleted == 'false' >>> >> >> > => the importer would add room3 as the record with the ID 2 and as >>> >> >> > consequence the room would be assigned to the organization 2 => >>> >> >> > which >>> >> >> > is wrong of course. >>> >> >>> >> >> > And the same you have between all tables. That is why I did import >>> >> >> > records always WITH all records marked as "deleted" AND in the >>> >> >> > import >>> >> >> > process I did add each object with the ID ALREADY SET. That means >>> >> >> > I >>> >> >> > did not use the generated Id from Hibernate, I did use the ID that >>> >> >> > was >>> >> >> > provided with the XML. But this does not work anymore as openJPA >>> >> >> > does >>> >> >> > not accept objects where the ID is > 0. >>> >> >> > At the moment everything is fine if you do a single import >>> >> >> > directly >>> >> >> > after the installation. But for example a user creates an >>> >> >> > Organization >>> >> >> > (will be Id = 2), deletes it again, the ID=2 is gone, if we do any >>> >> >> > import now with references to an organization=2 => everything >>> >> >> > Fails. >>> >> >> > In the old implementation we still where able to add an Org with >>> >> >> > ID=2 >>> >> >> > cause we did provide the ID in the Import Object and did not use >>> >> >> > the >>> >> >> > generated one for objects that are created using the import. >>> >> >>> >> >> > That is the problem, if you have an idea on what we might can do >>> >> >> > about >>> >> >> > those ID problem, maybe there is a Generator Class in openJPA that >>> >> >> > accepts self generated IDs. >>> >> >>> >> >> > Sebastian >>> >> >>> >> >> > 2011/7/18 Vasya <[email protected]>: >>> >> >>> >> >> > > Hi Sebastian, >>> >> >>> >> >> > > I have found and fixed two errors in backup: >>> >> >> > > 1. Roommanagement.getInstance().getBackupRooms() returns >>> >> >> > > error so >>> >> >> > > root >>> >> >> > > condition of criteria query is not defined. >>> >> >> > > 2. BackupImport.getAppointmentListByXML method does not >>> >> >> > > work >>> >> >> > > properly >>> >> >> > > because ‘user_id’ is not set. >>> >> >> > > I have fixed these errors and sent you email with patch. >>> >> >>> >> >> > > So, I have question about >>> >> >> > > UsersDaoImpl.getInstance().getAllUsersDeleted() method. It’s not >>> >> >> > > cleared for me what list of users should be returns all users or >>> >> >> > > deleted users? This method calls from BackupExport for export >>> >> >> > > users. >>> >> >> > > Please clarify what list of users should be return from this >>> >> >> > > method. >>> >> >>> >> >> > > Tomorrow I will continue to work on openmeetings and try to >>> >> >> > > resolve >>> >> >> > > other problems. >>> >> >>> >> >> > > Thanks, >>> >> >> > > Vasiliy. >>> >> >>> >> >> > > On Jul 18, 1:55 pm, "[email protected]" >>> >> >> > > <[email protected]> >>> >> >> > > wrote: >>> >> >> > >> Hi Vasya, >>> >> >>> >> >> > >> I think I've fixed it, but some more testing would be good. >>> >> >> > >> I also found out that sometimes emails contain the char:  >>> >> >> > >> (and >>> >> >> > >>  and  ) >>> >> >> > >> ... the SaxParser fails to read those files >>> >> >> > >> ... I think it is because people copy/paste their email from >>> >> >> > >> somewhere >>> >> >> > >> in the fields. >>> >> >> > >> But I have no idea about how they do it ... However this only >>> >> >> > >> happens >>> >> >> > >> 1 time per 1000 users but still strange. >>> >> >>> >> >> > >> For the Primary Key Issue I have no idea too, I changed it now >>> >> >> > >> but as >>> >> >> > >> you can imagine it can get quite confusing if the system >>> >> >> > >> generates >>> >> >> > >> the >>> >> >> > >> id itself and you import xxx records that "should" reference >>> >> >> > >> each >>> >> >> > >> other based on those IDs. For example rooms VS organisations or >>> >> >> > >> users >>> >> >> > >> VS organizations >>> >> >>> >> >> > >> Sebastian >>> >> >>> >> >> > >> 2011/7/18 Vasya <[email protected]>: >>> >> >>> >> >> > >> > Hello Sebastian, >>> >> >>> >> >> > >> > May be my test of backup was not completely. Today a have try >>> >> >> > >> > to >>> >> >> > >> > retest the import of backup more details. >>> >> >>> >> >> > >> > Thanks, >>> >> >>> >> >> > >> > Vasily. >>> >> >>> >> >> > >> > On Jul 18, 1:12 am, "[email protected]" >>> >> >> > >> > <[email protected]> >>> >> >> > >> > wrote: >>> >> >> > >> >> The import of Backup did not work. OpenJPA does not accept >>> >> >> > >> >> to add >>> >> >> > >> >> a >>> >> >> > >> >> new object where the Primary Key is already set. This is >>> >> >> > >> >> neccessary >>> >> >> > >> >> for importing existing Records from one database to another >>> >> >> > >> >> because of >>> >> >> > >> >> the foreign key relation-ships. >>> >> >>> >> >> > >> >> I have changed the code for now so that it works again. (in >>> >> >> > >> >> r3997) >>> >> >> > >> >> However, there are still some issues to be solved. >>> >> >>> >> >> > >> >> You might never run into that, the imported checks if the Id >>> >> >> > >> >> does >>> >> >> > >> >> exists before it adds a new record. So if you export a >>> >> >> > >> >> freshly >>> >> >> > >> >> installed openmeetings and then import it again into the >>> >> >> > >> >> same => >>> >> >> > >> >> Zero >>> >> >> > >> >> happens. >>> >> >> > >> >> So to test the backup import you really need to generate >>> >> >> > >> >> some >>> >> >> > >> >> users/conferencerooms/calendar-events/private messages and >>> >> >> > >> >> contacts et >>> >> >> > >> >> cetera and then trying to import that backup into a fresh >>> >> >> > >> >> installed >>> >> >> > >> >> openmeetings database. >>> >> >>> >> >> > >> >> Sebastian >>> >> >>> >> >> > >> >> 2011/7/16 [email protected] <[email protected]>: >>> >> >>> >> >> > >> >> > the saluation thing is fixed too.... >>> >> >>> >> >> > >> >> > Sebastian >>> >> >>> >> >> > >> >> > 2011/7/16 Maxim Solodovnik <[email protected]>: >>> >> >> > >> >> >> great to hear it :) >>> >> >> > >> >> >> please let me know in case of any issues you will found. >>> >> >>> >> >> > >> >> >> On Sat, Jul 16, 2011 at 20:27, [email protected] >>> >> >> > >> >> >> <[email protected]> >>> >> >> > >> >> >> wrote: >>> >> >>> >> >> > >> >> >>> ok ... >>> >> >>> >> >> > >> >> >>> I've fixed that issue with the order. Now I need to add >>> >> >> > >> >> >>> also >>> >> >> > >> >> >>> the where >>> >> >> > >> >> >>> clause in the annotation. This is really nice, this >>> >> >> > >> >> >>> manual >>> >> >> > >> >> >>> sub-select >>> >> >> > >> >> >>> for the main-menu always was a nasty hack because the >>> >> >> > >> >> >>> x-doclet >>> >> >> > >> >> >>> hbm.xml >>> >> >> > >> >> >>> creation did not allow those sub-queries with >>> >> >> > >> >> >>> where+orderby. >>> >> >> > >> >> >>> ... this >>> >> >> > >> >> >>> is also the reason why we have STRING flag *deleted* and >>> >> >> > >> >> >>> NOT a >>> >> >> > >> >> >>> boolean >>> >> >> > >> >> >>> field *deleted*. In previous hibernate/xdoclet creation >>> >> >> > >> >> >>> it was >>> >> >> > >> >> >>> not >>> >> >> > >> >> >>> possible to write a annotation with a boolean value as >>> >> >> > >> >> >>> where >>> >> >> > >> >> >>> clause ^^ >>> >> >> > >> >> >>> At some point we might clean that up too. >>> >> >>> >> >> > >> >> >>> The other one with the strang saluations_id / Primary >>> >> >> > >> >> >>> Key is >>> >> >> > >> >> >>> still to be >>> >> >> > >> >> >>> done. >>> >> >>> >> >> > >> >> >>> Sebastian >>> >> >>> >> >> > >> >> >>> 2011/7/16 [email protected] <[email protected]>: >>> >> >> > >> >> >>> > Hi Maxim, >>> >> >>> >> >> > >> >> >>> > I am currently checking out the openJPA refactoring. I >>> >> >> > >> >> >>> > like >>> >> >> > >> >> >>> > it, looks >>> >> >> > >> >> >>> > very nice and using annotations instead of XML files >>> >> >> > >> >> >>> > is >>> >> >> > >> >> >>> > quite elegant. >>> >> >>> >> >> > >> >> >>> > Maybe you have an idea about those things: >>> >> >> > >> >> >>> > - The navigation items are ordered different. For >>> >> >> > >> >> >>> > example by >>> >> >> > >> >> >>> > default >>> >> >> > >> >> >>> > you have the calendar instead of the dashboard now. >>> >> >> > >> >> >>> > And in >>> >> >> > >> >> >>> > the >>> >> >> > >> >> >>> > Administration you have a different order too. You >>> >> >> > >> >> >>> > have some >>> >> >> > >> >> >>> > code to >>> >> >> > >> >> >>> > do the one-to-many relation in your mapping. I think >>> >> >> > >> >> >>> > we >>> >> >> > >> >> >>> > might remove >>> >> >> > >> >> >>> > that and use java.util.List again, I think >>> >> >> > >> >> >>> > java.util.Set is >>> >> >> > >> >> >>> > not save >>> >> >> > >> >> >>> > in meaning of the order of items. >>> >> >> > >> >> >>> > - The table saluations start with the id 251 in my >>> >> >> > >> >> >>> > case. >>> >> >> > >> >> >>> > Normally >>> >> >> > >> >> >>> > record 1 should have salutations_id 1. That is >>> >> >> > >> >> >>> > necessary as >>> >> >> > >> >> >>> > the its >>> >> >> > >> >> >>> > hardcoded in the client adminstration panel that "Mr" >>> >> >> > >> >> >>> > has >>> >> >> > >> >> >>> > the ID 1. >>> >> >> > >> >> >>> > You got an idea about that? >>> >> >>> >> >> > >> >> >>> > Sebastian >>> >> >>> >> ... >>> >> >>> >> read more » >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> > Groups "OpenMeetings developers" group. >>> > To post to this group, send email to [email protected]. >>> > To unsubscribe from this group, send email to >>> > [email protected]. >>> > For more options, visit this group at >>> > http://groups.google.com/group/openmeetings-dev?hl=en. >>> > >>> > >>> >>> >>> >>> -- >>> Sebastian Wagner >>> http://www.webbase-design.de >>> http://openmeetings.googlecode.com >>> http://www.wagner-sebastian.com >>> [email protected] >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "OpenMeetings developers" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/openmeetings-dev?hl=en. >>> >> >> >> >> -- >> WBR >> Maxim aka solomax >> >> -- >> You received this message because you are subscribed to the Google Groups >> "OpenMeetings developers" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/openmeetings-dev?hl=en. >> > > > > -- > Sebastian Wagner > http://www.webbase-design.de > http://openmeetings.googlecode.com > http://www.wagner-sebastian.com > [email protected] > -- Sebastian Wagner http://www.webbase-design.de http://openmeetings.googlecode.com http://www.wagner-sebastian.com [email protected] -- You received this message because you are subscribed to the Google Groups "OpenMeetings developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/openmeetings-dev?hl=en.
