Guys, I wonder if it makes sense to switch to the java database at some point (for example Derby is a part of Apache). This would simplify installation now, and at some point may result in better JIT optimizations.
-- With best regards / с наилучшими пожеланиями, Alexei Fedotov / Алексей Федотов, http://dataved.ru/ +7 916 562 8095 On Thu, Jul 21, 2011 at 7:45 PM, [email protected] <[email protected]> wrote: > Yeah, > > you're right. I would suggest if it works with a 40.000 Users > import/export, at lets say -Xmx512MB for the hole red5 server process, > I think its an acceptable performance. > > Sebastian > > 2011/7/21 Maxim Solodovnik <[email protected]>: >> If memory used amount of memory will be unacceptably high we can use >> temporary tables for store id pairs. >> With openJPA it will be just couple additional beans. >> >> >> On Thu, Jul 21, 2011 at 16:21, [email protected] <[email protected]> >> wrote: >>> >>> Hi Maxim, >>> >>> yes this was the thing that I tried to avoid :) Collecting all IDs and >>> building lookup tables to see which old id corresponds to the record >>> after the import. >>> >>> The other thing, to import the objects in the correct order is already >>> implemented. >>> >>> Some Issue will be also the uploaded files: There are some >>> restrictions on the user's profile images. I think they have the >>> userId as folderName in the path. So you can't copy the folders >>> anymore into the disk just by coping the ROOT folder of the >>> upload-directory from the Backup-ZIP to the disk, cause there are some >>> subfolders that contain invalid folderNames that need to be >>> rewritten/renamed. >>> >>> We can try that out, its obviously better if it works but its a rather >>> big change. >>> My concern is the same like yours: Could be quite memory hungry process. >>> >>> I got for example a database with 40.000++ users that I need to >>> import/export. Holding all 40.000 users would not work, maybe holding >>> the 40.000 IDs could work. >>> >>> Sebastian >>> >>> 2011/7/21 Maxim Solodovnik <[email protected]>: >>> > Hello Sebastian, Vasiliy >>> > Instead of creating generator classes I would like to propose the >>> > following >>> > mechanism: >>> > 1) Import should be done in "well known" order i.e. organisanisations >>> > are >>> > imported first then users, then rooms etc. >>> > 2) While importing certain object id is not set. Instead of this old id >>> > and >>> > new id are stored in hashtable key== oldId, value == newId >>> > 3) While importing objects containing links to "parent objects" newId is >>> > used using Hashtable created. >>> > 4) Pictures/streams/videos are stored in the file system using newId >>> > The import will be memory consuming operation but all constraints will >>> > be >>> > valid. >>> > >>> > On Mon, Jul 18, 2011 at 17:46, Vasya <[email protected]> wrote: >>> >> >>> >> Hi Sebastian, >>> >> >>> >> Thanks for your answer! >>> >> >>> >> I will continue to investigate problem tomorrow. >>> >> >>> >> Vasiliy. >>> >> >>> >> >>> >> On Jul 18, 5:32 pm, "[email protected]" <[email protected]> >>> >> wrote: >>> >> > Hi Vasya, >>> >> > >>> >> > If tested getBackupRooms yesterday, it was okay for me. >>> >> > >>> >> > The thing with the *deleted* is that it has to return ALL records, no >>> >> > matter if the deleted flag is set or not. >>> >> > This is the case because of the foreign key references. >>> >> > >>> >> > Just imagine the following sample: >>> >> > Rooms-Table: >>> >> > ID name deleted >>> >> > 1 room1 false >>> >> > 2 room2 true >>> >> > 3 room3 false >>> >> > >>> >> > Rooms-Organizations Table: >>> >> > ID rooms_id organisation_id >>> >> > 1 1 1 >>> >> > 1 2 2 >>> >> > 1 3 1 >>> >> > >>> >> > If you only export those records that are deleted == 'false' >>> >> > => the importer would add room3 as the record with the ID 2 and as >>> >> > consequence the room would be assigned to the organization 2 => which >>> >> > is wrong of course. >>> >> > >>> >> > And the same you have between all tables. That is why I did import >>> >> > records always WITH all records marked as "deleted" AND in the import >>> >> > process I did add each object with the ID ALREADY SET. That means I >>> >> > did not use the generated Id from Hibernate, I did use the ID that >>> >> > was >>> >> > provided with the XML. But this does not work anymore as openJPA does >>> >> > not accept objects where the ID is > 0. >>> >> > At the moment everything is fine if you do a single import directly >>> >> > after the installation. But for example a user creates an >>> >> > Organization >>> >> > (will be Id = 2), deletes it again, the ID=2 is gone, if we do any >>> >> > import now with references to an organization=2 => everything Fails. >>> >> > In the old implementation we still where able to add an Org with ID=2 >>> >> > cause we did provide the ID in the Import Object and did not use the >>> >> > generated one for objects that are created using the import. >>> >> > >>> >> > That is the problem, if you have an idea on what we might can do >>> >> > about >>> >> > those ID problem, maybe there is a Generator Class in openJPA that >>> >> > accepts self generated IDs. >>> >> > >>> >> > Sebastian >>> >> > >>> >> > 2011/7/18 Vasya <[email protected]>: >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > > Hi Sebastian, >>> >> > >>> >> > > I have found and fixed two errors in backup: >>> >> > > 1. Roommanagement.getInstance().getBackupRooms() returns error >>> >> > > so >>> >> > > root >>> >> > > condition of criteria query is not defined. >>> >> > > 2. BackupImport.getAppointmentListByXML method does not work >>> >> > > properly >>> >> > > because 'user_id' is not set. >>> >> > > I have fixed these errors and sent you email with patch. >>> >> > >>> >> > > So, I have question about >>> >> > > UsersDaoImpl.getInstance().getAllUsersDeleted() method. It's not >>> >> > > cleared for me what list of users should be returns all users or >>> >> > > deleted users? This method calls from BackupExport for export >>> >> > > users. >>> >> > > Please clarify what list of users should be return from this >>> >> > > method. >>> >> > >>> >> > > Tomorrow I will continue to work on openmeetings and try to resolve >>> >> > > other problems. >>> >> > >>> >> > > Thanks, >>> >> > > Vasiliy. >>> >> > >>> >> > > On Jul 18, 1:55 pm, "[email protected]" <[email protected]> >>> >> > > wrote: >>> >> > >> Hi Vasya, >>> >> > >>> >> > >> I think I've fixed it, but some more testing would be good. >>> >> > >> I also found out that sometimes emails contain the char:  (and >>> >> > >>  and  ) >>> >> > >> ... the SaxParser fails to read those files >>> >> > >> ... I think it is because people copy/paste their email from >>> >> > >> somewhere >>> >> > >> in the fields. >>> >> > >> But I have no idea about how they do it ... However this only >>> >> > >> happens >>> >> > >> 1 time per 1000 users but still strange. >>> >> > >>> >> > >> For the Primary Key Issue I have no idea too, I changed it now but >>> >> > >> as >>> >> > >> you can imagine it can get quite confusing if the system generates >>> >> > >> the >>> >> > >> id itself and you import xxx records that "should" reference each >>> >> > >> other based on those IDs. For example rooms VS organisations or >>> >> > >> users >>> >> > >> VS organizations >>> >> > >>> >> > >> Sebastian >>> >> > >>> >> > >> 2011/7/18 Vasya <[email protected]>: >>> >> > >>> >> > >> > Hello Sebastian, >>> >> > >>> >> > >> > May be my test of backup was not completely. Today a have try to >>> >> > >> > retest the import of backup more details. >>> >> > >>> >> > >> > Thanks, >>> >> > >>> >> > >> > Vasily. >>> >> > >>> >> > >> > On Jul 18, 1:12 am, "[email protected]" >>> >> > >> > <[email protected]> >>> >> > >> > wrote: >>> >> > >> >> The import of Backup did not work. OpenJPA does not accept to >>> >> > >> >> add >>> >> > >> >> a >>> >> > >> >> new object where the Primary Key is already set. This is >>> >> > >> >> neccessary >>> >> > >> >> for importing existing Records from one database to another >>> >> > >> >> because of >>> >> > >> >> the foreign key relation-ships. >>> >> > >>> >> > >> >> I have changed the code for now so that it works again. (in >>> >> > >> >> r3997) >>> >> > >> >> However, there are still some issues to be solved. >>> >> > >>> >> > >> >> You might never run into that, the imported checks if the Id >>> >> > >> >> does >>> >> > >> >> exists before it adds a new record. So if you export a freshly >>> >> > >> >> installed openmeetings and then import it again into the same >>> >> > >> >> => >>> >> > >> >> Zero >>> >> > >> >> happens. >>> >> > >> >> So to test the backup import you really need to generate some >>> >> > >> >> users/conferencerooms/calendar-events/private messages and >>> >> > >> >> contacts et >>> >> > >> >> cetera and then trying to import that backup into a fresh >>> >> > >> >> installed >>> >> > >> >> openmeetings database. >>> >> > >>> >> > >> >> Sebastian >>> >> > >>> >> > >> >> 2011/7/16 [email protected] <[email protected]>: >>> >> > >>> >> > >> >> > the saluation thing is fixed too.... >>> >> > >>> >> > >> >> > Sebastian >>> >> > >>> >> > >> >> > 2011/7/16 Maxim Solodovnik <[email protected]>: >>> >> > >> >> >> great to hear it :) >>> >> > >> >> >> please let me know in case of any issues you will found. >>> >> > >>> >> > >> >> >> On Sat, Jul 16, 2011 at 20:27, [email protected] >>> >> > >> >> >> <[email protected]> >>> >> > >> >> >> wrote: >>> >> > >>> >> > >> >> >>> ok ... >>> >> > >>> >> > >> >> >>> I've fixed that issue with the order. Now I need to add >>> >> > >> >> >>> also >>> >> > >> >> >>> the where >>> >> > >> >> >>> clause in the annotation. This is really nice, this manual >>> >> > >> >> >>> sub-select >>> >> > >> >> >>> for the main-menu always was a nasty hack because the >>> >> > >> >> >>> x-doclet >>> >> > >> >> >>> hbm.xml >>> >> > >> >> >>> creation did not allow those sub-queries with >>> >> > >> >> >>> where+orderby. >>> >> > >> >> >>> ... this >>> >> > >> >> >>> is also the reason why we have STRING flag *deleted* and >>> >> > >> >> >>> NOT a >>> >> > >> >> >>> boolean >>> >> > >> >> >>> field *deleted*. In previous hibernate/xdoclet creation it >>> >> > >> >> >>> was >>> >> > >> >> >>> not >>> >> > >> >> >>> possible to write a annotation with a boolean value as >>> >> > >> >> >>> where >>> >> > >> >> >>> clause ^^ >>> >> > >> >> >>> At some point we might clean that up too. >>> >> > >>> >> > >> >> >>> The other one with the strang saluations_id / Primary Key >>> >> > >> >> >>> is >>> >> > >> >> >>> still to be >>> >> > >> >> >>> done. >>> >> > >>> >> > >> >> >>> Sebastian >>> >> > >>> >> > >> >> >>> 2011/7/16 [email protected] <[email protected]>: >>> >> > >> >> >>> > Hi Maxim, >>> >> > >>> >> > >> >> >>> > I am currently checking out the openJPA refactoring. I >>> >> > >> >> >>> > like >>> >> > >> >> >>> > it, looks >>> >> > >> >> >>> > very nice and using annotations instead of XML files is >>> >> > >> >> >>> > quite elegant. >>> >> > >>> >> > >> >> >>> > Maybe you have an idea about those things: >>> >> > >> >> >>> > - The navigation items are ordered different. For example >>> >> > >> >> >>> > by >>> >> > >> >> >>> > default >>> >> > >> >> >>> > you have the calendar instead of the dashboard now. And >>> >> > >> >> >>> > in >>> >> > >> >> >>> > the >>> >> > >> >> >>> > Administration you have a different order too. You have >>> >> > >> >> >>> > some >>> >> > >> >> >>> > code to >>> >> > >> >> >>> > do the one-to-many relation in your mapping. I think we >>> >> > >> >> >>> > might remove >>> >> > >> >> >>> > that and use java.util.List again, I think java.util.Set >>> >> > >> >> >>> > is >>> >> > >> >> >>> > not save >>> >> > >> >> >>> > in meaning of the order of items. >>> >> > >> >> >>> > - The table saluations start with the id 251 in my case. >>> >> > >> >> >>> > Normally >>> >> > >> >> >>> > record 1 should have salutations_id 1. That is necessary >>> >> > >> >> >>> > as >>> >> > >> >> >>> > the its >>> >> > >> >> >>> > hardcoded in the client adminstration panel that "Mr" has >>> >> > >> >> >>> > the ID 1. >>> >> > >> >> >>> > You got an idea about that? >>> >> > >>> >> > >> >> >>> > Sebastian >>> >> > >>> >> > >> >> >>> > 2011/7/16 Maxim Solodovnik <[email protected]>: >>> >> > >> >> >>> >> Hello Sebastian, >>> >> > >> >> >>> >> I just checked: the last extension listed is png >>> >> > >> >> >>> >> and it is available for selection. >>> >> > >>> >> > >> >> >>> >> On Sat, Jul 16, 2011 at 17:58, [email protected] >>> >> > >> >> >>> >> <[email protected]> >>> >> > >> >> >>> >> wrote: >>> >> > >>> >> > >> >> >>> >>> Have you verified that all extension are visible then >>> >> > >> >> >>> >>> in >>> >> > >> >> >>> >>> the windows >>> >> > >> >> >>> >>> upload dialog? >>> >> > >> >> >>> >>> In previous flash version after xx extension it just >>> >> > >> >> >>> >>> cuts >>> >> > >> >> >>> >>> off and the >>> >> > >> >> >>> >>> rest was not visible or selectable at all. That was >>> >> > >> >> >>> >>> weird. >>> >> > >> >> >>> >>> That is why >>> >> > >> >> >>> >>> there are multiple columns and possibilities to choose. >>> >> > >>> >> > >> >> >>> >>> Sebastian >>> >> > >>> >> > >> >> >>> >>> 2011/7/16 <[email protected]>: >>> >> > >> >> >>> >>> > Revision: 3988 >>> >> > >> >> >>> >>> > Author: solomax666 >>> >> > >> >> >>> >>> > Date: Sat Jul 16 02:59:13 2011 >>> >> > >> >> >>> >>> > Log: - "Allowed Files" should contain all >>> >> > >> >> >>> >>> > extensions allowed >>> >> > >> >> >>> >>> >>> >> > >> >> >>> >>> > >>> >> > >> >> >>> >>> > > >http://code.google.com/p/openmeetings/source/detail?r=3988 >>> >> > >>> >> > >> >> >>> >>> > Modified: >>> >> > >>> >> > >> >> >>> >>> > >>> >> > >> >> >>> >>> > >>> >> > >> >> >>> >>> > >>> >> > >> >> >>> >>> > /trunk/singlewebapp/WebContent/openmeetings/base/components/upload/fileUpload.lzx >>> >> > >>> >> > >> >> >>> >>> > ======================================= >>> >> > >> >> >>> >>> > --- >>> >> > >>> >> > >> >> >>> >>> > >>> >> > >> >> >>> >>> > >>> >> > >> >> >>> >>> > /trunk/singlewebapp/WebContent/openmeetings/base/components/upload/fileUpload.lzx >>> >> > >> >> >>> >>> > Fri Jun 24 08:16:56 2011 >>> >> > >> >> >>> >>> > +++ >>> >> > >>> >> > >> >> >>> >>> > >>> >> > >> >> >>> >>> > >>> >> > >> >> >>> >>> > /trunk/singlewebapp/WebContent/openmeetings/base/components/upload/fileUpload.lzx >>> >> > >> >> >>> >>> > Sat Jul 16 02:59:13 2011 >>> >> > >> >> >>> >>> > @@ -64,7 +64,7 @@ >>> >> > >>> >> > >> >> >>> >>> > var imageTypes = new >>> >> > >> >> >>> >>> > Object(); >>> >> > >> >> >>> >>> > imageTypes.description >>> >> > >> >> >>> >>> > = >>> >> > >> >> >>> >>> > "Allowed >>> >> > >> >> >>> >>> > Files"; >>> >> > >> >> >>> >>> > - imageTypes.extension >>> >> > >> >> >>> >>> > = >>> >> > >> >> >>> >>> > "*.txt; >>> >> > >> >> >>> >>> > *.pcd; >>> >> > >> >> >>> >>> > *.ps; >>> >> > >> >> >>> >>> > *.psd; *.tiff; *.bmp; *.svg; *.jpg; *.jpeg; *.gif; >>> >> > >> >> >>> >>> > *.png; *.ppt; >>> >> > >> >> >>> >>> > *.pptx; >>> >> > >> >> >>> >>> > *.odp; *.odt; *.doc; *.docx; *.rtf; *.txt; *.ods; >>> >> > >> >> >>> >>> > *.sxc; >>> >> > >> >> >>> >>> > *.xls; >>> >> > >> >> >>> >>> > *.xlsx; >>> >> > >> >> >>> >>> > *.pdf"; >>> >> > >> >> >>> >>> > + imageTypes.extension >>> >> > >> >> >>> >>> > = >>> >> > >> >> >>> >>> > "*.txt; >>> >> > >> >> >>> >>> > *.ppt; >>> >> > >> >> >>> >>> > *.pptx; *.odp; *.odt; *.wpd; *.doc; *.docx; *.rtf; >>> >> > >> >> >>> >>> > *.txt; *.ods; >>> >> > >> >> >>> >>> > *.sxc; >>> >> > >> >> >>> >>> > *.xls; *.xlsx; *.sxi; *.pdf; *.avi; *.mov; *.flv; >>> >> > >> >> >>> >>> > *.mp4; >>> >> > >> >> >>> >>> > *.tga; >>> >> > >> >> >>> >>> > *.xcf; >>> >> > >> >> >>> >>> > *.wpg; *.ico; *.ttf; *.pcd; *.pcds; *.ps; *.psd; >>> >> > >> >> >>> >>> > *.tiff; >>> >> > >> >> >>> >>> > *.bmp; >>> >> > >> >> >>> >>> > *.svg; >>> >> > >> >> >>> >>> > *.dpx; *.exr; *.jpg; *.jpeg; *.gif; *.png"; >>> >> > >> >> >>> >>> > >>> >> > >> >> >>> >>> > allTypes.push(imageTypes); >>> >> > >>> >> > >> >> >>> >>> > imageTypes = new >>> >> > >> >> >>> >>> > Object(); >>> >> > >> >> >>> >>> > @@ -78,7 +78,7 @@ >>> >> > >> >> >>> >>> > allTypes.push(imageTypes); >>> >> > >>> >> > >> >> >>> >>> > imageTypes = new >>> >> > >> >> >>> >>> > Object(); >>> >> > >> >> >>> >>> > - >>> >> > >>> >> > ... >>> >> > >>> >> > read more >> >>> >> >>> >> -- >>> >> You received this message because you are subscribed to the Google >>> >> Groups >>> >> "OpenMeetings developers" group. >>> >> To post to this group, send email to [email protected]. >>> >> To unsubscribe from this group, send email to >>> >> [email protected]. >>> >> For more options, visit this group at >>> >> http://groups.google.com/group/openmeetings-dev?hl=en. >>> >> >>> > >>> > >>> > >>> > -- >>> > WBR >>> > Maxim aka solomax >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> > Groups >>> > "OpenMeetings developers" group. >>> > To post to this group, send email to [email protected]. >>> > To unsubscribe from this group, send email to >>> > [email protected]. >>> > For more options, visit this group at >>> > http://groups.google.com/group/openmeetings-dev?hl=en. >>> > >>> >>> >>> >>> -- >>> Sebastian Wagner >>> http://www.webbase-design.de >>> http://openmeetings.googlecode.com >>> http://www.wagner-sebastian.com >>> [email protected] >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "OpenMeetings developers" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/openmeetings-dev?hl=en. >>> >> >> >> >> -- >> WBR >> Maxim aka solomax (icq #20779631; skype maxim.solodovnik) >> mobile phone: +79133861155 >> >> -- >> You received this message because you are subscribed to the Google Groups >> "OpenMeetings developers" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/openmeetings-dev?hl=en. >> > > > > -- > Sebastian Wagner > http://www.webbase-design.de > http://openmeetings.googlecode.com > http://www.wagner-sebastian.com > [email protected] > > -- > You received this message because you are subscribed to the Google Groups > "OpenMeetings developers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/openmeetings-dev?hl=en. > > -- You received this message because you are subscribed to the Google Groups "OpenMeetings developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/openmeetings-dev?hl=en.
