Re: [Zope] Should I use ZODB or not ?
Jason Cunliffe writes: > OK, sorry to be so dense about this, but when you say "through ZServer" > ...please clarify for me on thing: > > Does LocalFS actually go "through ZServer" ? > I thought that all it does, is to make a virtual link, storing a pointer in > the ZODB to the real physical file system. > Upload thus via FTP would be as fast as could be, and download could be > defined if you prefer FTP or HTTP. > Does'nt the link in LocalFS work the same in both directions, merely storing > the linkn in XODB adn pass the location to the protocol? It does. When I stress that you should not go through ZServer, it means do not upload/serve via Zope. Use FTP or another protocol appropriate for mass data transfer over unstable connections. Dieter ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Should I use ZODB or not ?
Dieter Thanks for your answer and advice.. > > I am curious how do you all think LocalFS would perform for such a big > > multimedia application.? > > http://www.zope.org/Members/jfarr/Products/LocalFS > In my view, LocalFS is an excellent product. I like it very much. > > However, I would not upload and not serve objects with a size > of several hundred megabytes through ZServer (and thereby through > Zope). > As I said in an earlier response, ZServer is at least on > order of magnitude slower with static content than > Apache. This matters, if you transfer big objects! OK, sorry to be so dense about this, but when you say "through ZServer" ..please clarify for me on thing: Does LocalFS actually go "through ZServer" ? I thought that all it does, is to make a virtual link, storing a pointer in the ZODB to the real physical file system. Upload thus via FTP would be as fast as could be, and download could be defined if you prefer FTP or HTTP. Does'nt the link in LocalFS work the same in both directions, merely storing the linkn in XODB adn pass the location to the protocol? > Recently, there has been some simple benchmarks for > Zope versus Apache performance in this list. > Search the archive (there is a searchable one at NIP) > to get a feeling about the bandwidth. > > You may well use LocalFS to view the metadata (id, size, modification time, > ...) from Zope, but you should not go through ZServer for upload > and HTTP requests to really large objects. Again I dont understand. With LocalFS, isn't the "really large object" outside of Zope. Surely it does not care how big the file is? All it cares about is its location, and any extra metadata IF you choose to store it in Zope. yes / no? thanks - Jason ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Should I use ZODB or not ?
Jason Cunliffe writes: > I am curious how do you all think LocalFS would perform for such a big > multimedia application.? > http://www.zope.org/Members/jfarr/Products/LocalFS In my view, LocalFS is an excellent product. I like it very much. However, I would not upload and not serve objects with a size of several hundred megabytes through ZServer (and thereby through Zope). As I said in an earlier response, ZServer is at least on order of magnitude slower with static content than Apache. This matters, if you transfer big objects! Recently, there has been some simple benchmarks for Zope versus Apache performance in this list. Search the archive (there is a searchable one at NIP) to get a feeling about the bandwidth. You may well use LocalFS to view the metadata (id, size, modification time, ...) from Zope, but you should not go through ZServer for upload and HTTP requests to really large objects. Dieter ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Should I use ZODB or not ?
On Tue, 19 Sep 2000, Jason Cunliffe wrote: > I am curious how do you all think LocalFS would perform for such a big > multimedia application.? > http://www.zope.org/Members/jfarr/Products/LocalFS > > Use Zope for managing meta-data, or mayeb even better Zope + a fast DB tool > for the metadata part. > Let Zope do its holy best for the 'virtual-object interfacing' jobs, let the > others do their thing.. This may very well be the best solution, as far as I understand what LocalFS is (I've not used it for now, however), because this may allow us to set per-domain (Medicine, Maths, etc...) permissions under Zope, and do all the presentation in Zope, while not being attached to Zope if we want to easily dump our datas somewhere else. Thanks to all. Now all I have to do is just kill these PHP guys there ;-)) bye, Jerome Alet ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Should I use ZODB or not ?
Hello I am curious how do you all think LocalFS would perform for such a big multimedia application.? http://www.zope.org/Members/jfarr/Products/LocalFS Use Zope for managing meta-data, or mayeb even better Zope + a fast DB tool for the metadata part. Let Zope do its holy best for the 'virtual-object interfacing' jobs, let the others do their thing.. My understanding was that using the LocalFS is as fast as you can get really and that any file size limits are to do with the OS, disks RAM, caching etc, not Zope or anything else. Plus one is free to scale or mix and match zope with other tools and systems, as it develops without compromising all the work which has gone into the base data's filestructure. Might be much easier to develop too.. The data would not be locked into Zope or for that matter any RDBMS 'solution'. One coud performance tests, develop optimizations and determin along the way when and where to apply them. No headaches as Zope goes through upgrades either because one is not taxing it too much. Can anyone see what are the faults with this approach? I have been wondering about such what plan to use and if or with Zope for large animated map datasets project. This would use Flash SWF, and also SVG [XMLbased Scalable Vector Graphics] The SWF can be merged into Quicktime allowing for multiple tracks. This is a powerful combinatoin where one needs to package trainging, presentation. Video though big contains a huge amount of 'information' which is lost in all other infromation system: body language, human language, tone of voice, inflexion, accent, emphasis, mood, lighting, context, scale, enviromnental conditions. etc. - Jason Jason CUNLIFFE = NOMADICS.(Interactive Art and Technology).Design Director - Original Message - From: Dieter Maurer <[EMAIL PROTECTED]> To: Jerome Alet <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Tuesday, September 19, 2000 3:44 PM Subject: Re: [Zope] Should I use ZODB or not ? > Jerome Alet writes: > > Our needs are to store hundreds, and possibly thousands of MPEG2 and MPEG4 > > (generally short) video movies, MP3 audio files, JPEG images, etc... > > > > Altough most of the files will be in the 1 Mb -> 10 Mb range, some could > > be larger, e.g. some Gigabytes. > > > > For each file we also want to keep some data: author, subject, encoding, > > description, keywords, domain, etc... > > > > I see 5 solutions: > > > > 1 - store both files and their associated datas in ZODB > > 2 - store files in the filesystem and datas in ZODB > > 3 - store files in the filesystem and datas in an RDBMS > > (use Zope only for the presentation of files and datas) > > 4 - store files in ZODB and datas in an RDBMS. > > 5 - don't use Zope at all because it's not appropriate. > > Do not store the files in Zope or serve them with Zope. > A standard web server (e.g. Apache) is at least one order > of magnitude faster than Zope for static content (such as > multi media files). This is essential for you > large file objects. > > If you have up to a few thousand objects and they do not change too > often, you can put them (the meta data) into the ZODB. > Otherwise, put them in a RDBMS. > I would use the RDBMS whenever I have one or I have more than > a few hundred objects or I would need access to the data > from legacy processes, too. > > > Dieter > > ___ > Zope maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope-dev ) > > ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Should I use ZODB or not ?
Jerome Alet writes: > Our needs are to store hundreds, and possibly thousands of MPEG2 and MPEG4 > (generally short) video movies, MP3 audio files, JPEG images, etc... > > Altough most of the files will be in the 1 Mb -> 10 Mb range, some could > be larger, e.g. some Gigabytes. > > For each file we also want to keep some data: author, subject, encoding, > description, keywords, domain, etc... > > I see 5 solutions: > > 1 - store both files and their associated datas in ZODB > 2 - store files in the filesystem and datas in ZODB > 3 - store files in the filesystem and datas in an RDBMS > (use Zope only for the presentation of files and datas) > 4 - store files in ZODB and datas in an RDBMS. > 5 - don't use Zope at all because it's not appropriate. Do not store the files in Zope or serve them with Zope. A standard web server (e.g. Apache) is at least one order of magnitude faster than Zope for static content (such as multi media files). This is essential for you large file objects. If you have up to a few thousand objects and they do not change too often, you can put them (the meta data) into the ZODB. Otherwise, put them in a RDBMS. I would use the RDBMS whenever I have one or I have more than a few hundred objects or I would need access to the data from legacy processes, too. Dieter ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Should I use ZODB or not ?
From: "Jerome Alet" <[EMAIL PROTECTED]> [snip] > Our needs are to store hundreds, and possibly thousands of MPEG2 and MPEG4 > (generally short) video movies, MP3 audio files, JPEG images, etc... > [snip] > > I see 5 solutions: > > 1 - store both files and their associated datas in ZODB > 2 - store files in the filesystem and datas in ZODB > 3 - store files in the filesystem and datas in an RDBMS > (use Zope only for the presentation of files and datas) > 4 - store files in ZODB and datas in an RDBMS. > 5 - don't use Zope at all because it's not appropriate. > In general, large file uploads thru HTTP are clumsy. The browsers don't provide progress indicators. Broken connections occur due to timeout under network congestion but the browsers don't handle broken connections well. And (as far as I can tell) the entire request (including the entire file) is buffered in the OS swapspace. Some site I saw recently (was it xdrive? I'm not sure) uses client-side Java to manage the upload. I chose #2 for a similar application and ran into these additional issues: 1. A file upload done thru Zope into the filesystem passes thru a temporary file. This increases slightly the apparent upload time, while you copy the file to its final destination. And, you must also ensure that the partition where temporary files are placed has enough space for the maximum file size. 2. You must also decide how you are going to serve the file. Since it's in the file system, you could serve it directly thru Apache (perhaps thru a redirect initiated by Zope), thereby avoiding occupying a Zope thread for a long time for every downlaod. But if you take this approach, you must somehow make Apache apply the same access control to the file that you have in Zope. This can be done, but is an added complication. -- My .02 -- Loren ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
[Zope] Should I use ZODB or not ?
Hi, we are currently planning to create a multimedia database and are investigating different solutions. I'm pushing toward a Zope based one, especially for the permissions needs we have, but I was wondering if ZODB can handle our needs or if we should store multimedia files directly on the file system. Our needs are to store hundreds, and possibly thousands of MPEG2 and MPEG4 (generally short) video movies, MP3 audio files, JPEG images, etc... Altough most of the files will be in the 1 Mb -> 10 Mb range, some could be larger, e.g. some Gigabytes. For each file we also want to keep some data: author, subject, encoding, description, keywords, domain, etc... I see 5 solutions: 1 - store both files and their associated datas in ZODB 2 - store files in the filesystem and datas in ZODB 3 - store files in the filesystem and datas in an RDBMS (use Zope only for the presentation of files and datas) 4 - store files in ZODB and datas in an RDBMS. 5 - don't use Zope at all because it's not appropriate. While I'm pretty sure I can design something more or less working like the 1 above in a few time with a few multimedia files and datas, please could you tell me what seems the best solution to you in the long run ? IMHO the 1 (if Zope scales well) and 2 are best, but we may as well use the 3. 4 doesn't seem natural to me, and I personnaly exclude the 5 ;-) Another question I wanted to ask: does Zope have a limit on the size of uploaded files, given the fact that we may want to upload files as large as some Gb and that we will use a linux filesystem not limited to 2 Gb per file. Please show me the light. thanks in advance. Jerome ALET - [EMAIL PROTECTED] - http://cortex.unice.fr/~jerome Faculte de Medecine de Nice - http://noe.unice.fr - Tel: 04 93 37 76 30 28 Avenue de Valombrose - 06107 NICE Cedex 2 - FRANCE ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
