RE: mountSharedResource() on huge amount of images
> -Original Message- > From: mzem...@osc.state.ny.us [mailto:mzem...@osc.state.ny.us] > Sent: Tuesday, March 15, 2011 7:28 PM > To: users@wicket.apache.org > Subject: Re: mountSharedResource() on huge amount of images > > No offense Bas but that seems like a major hassle, especially > considering with an ORM its a simple update/delete/find. > When it comes time to build out new servers now you have to > shuffle around 300k photos instead of simply replicating a > database. Also sounds like it would make debugging more > difficult when your images are three or more folders deep. > And what if you want to store attributes along with the > photos? Like say user comments, or flag them for > inappropriate content, copyright infringement, etc. What if > there is an open file handle when you try to delete the > image? Just seems to me a much smoother solution to put in a > db. But with that I'm bowing out because as you said it's a > heated debate and comes down to personal preference. Next version will have images in db. Arguments you serve I know, from point of another atributes database is better. > > > > > From: Bas Gooren > To: users@wicket.apache.org > Date: 03/15/2011 02:10 PM > Subject:Re: mountSharedResource() on huge amount of images > > > > Now you're talking about rendering them, which is a different > topic than mounting a resource which serves said images. > > You'll only need to mount a single shared resource which > serves all the images. However, given the amount of images > you can consider allowing your front-end (e.g. apache httd) > or a dedicated webserver serve the images. > > Since you mention that the amount of images can be > potentially large (>250), I'd suggest removing the "show all" > option, or using an ajax viewport (max 20-50 images on-screen > at a time, when the user scrolls you load new images over ajax). > > Someone else just suggested storing images in the database. > While there is usually heated debate about this topic (files > on disk vs in the database), let me just say that simply > having lots of images is no reason to move images into the > database. For starters, you can always store your files in a > hashed folder structure, e.g. when the ID is 1234, store the > image in a file/folder called /1/2/3/4.jpg > > Bas > > Op 15-3-2011 18:00, Ladislav DANKO schreef: > > Imagine this situation: users have accounts on photo album > where they > upload > > images. System from uploaded images create thumbnails. Users can > > browse their photo - there is combo "show 25", "show 50"... > "show all" images. > On > > page > > I render thumbnails on a page which are shadowbox clickable images. > > All images ("show all") I do in way described below. > > Works fine but in extreme situation there is user with more > than 3.000 > > images in one photoalbum. > > > > Or -how to do it better way? > > > > Thanks, > > > > Laco > > > > > > > >> -Original Message- > >> From: Bas Gooren [mailto:b...@iswd.nl] > >> Sent: Thursday, March 10, 2011 11:32 PM > >> To: users@wicket.apache.org > >> Subject: Re: mountSharedResource() on huge amount of images > >> > >> The general idea is to mount a single handler, which takes the > >> filename from the url. > >> There is no reason to mount all images by such a handler > one-by-one. > >> > >> Bas > >> > >> Op 10-3-2011 23:01, Ladislav DANKO schreef: > >>> Hi all, > >>> > >>> what is the recommended way to mount huge amount of an images > >>> (thousands) in app? Does mountSharedResource() has any bottleneck? > >>> Simplified code look > >>> like: > >>> > >>> Folder folder = ((Start) Application.get()).getUploadFolder(); > >>> File[] files = folder.getFiles(); > >>> List lList = Arrays.asList(files); > >> Collections.sort(lList); int > >>> i = 0; Iterator iterator = lList.iterator(); > >>> while(iterator.hasNext()) > >>> { > >>> iterator.next(); > >>> String fileName = lList.get(i).getName(); > >>> mountSharedResource("/images/" + fileName, new > >>> ResourceReference(Images.class, fileName).getSharedResourceKey()); > >>> i++; > >>> } > >>> > >>> But what if in fol
RE: mountSharedResource() on huge amount of images
> -Original Message- > From: Bas Gooren [mailto:b...@iswd.nl] > Sent: Tuesday, March 15, 2011 7:10 PM > To: users@wicket.apache.org > Subject: Re: mountSharedResource() on huge amount of images > > Now you're talking about rendering them, which is a different > topic than mounting a resource which serves said images. > > You'll only need to mount a single shared resource which > serves all the images. However, given the amount of images > you can consider allowing your front-end (e.g. apache httd) > or a dedicated webserver serve the images. Is there example, please? > > Since you mention that the amount of images can be > potentially large (>250), I'd suggest removing the "show all" > option, or using an ajax viewport (max 20-50 images on-screen > at a time, when the user scrolls you load new images over ajax). > > Someone else just suggested storing images in the database. > While there is usually heated debate about this topic (files > on disk vs in the database), let me just say that simply > having lots of images is no reason to move images into the > database. For starters, you can always store your files in a > hashed folder structure, e.g. when the ID is 1234, store the > image in a file/folder called /1/2/3/4.jpg > > Bas > > Op 15-3-2011 18:00, Ladislav DANKO schreef: > > Imagine this situation: users have accounts on photo album > where they > > upload images. System from uploaded images create thumbnails. Users > > can browse their photo - there is combo "show 25", "show > 50"... "show > > all" images. On page I render thumbnails on a page which > are shadowbox > > clickable images. > > All images ("show all") I do in way described below. > > Works fine but in extreme situation there is user with more > than 3.000 > > images in one photoalbum. > > > > Or -how to do it better way? > > > > Thanks, > > > > Laco > > > > > > > >> -Original Message- > >> From: Bas Gooren [mailto:b...@iswd.nl] > >> Sent: Thursday, March 10, 2011 11:32 PM > >> To: users@wicket.apache.org > >> Subject: Re: mountSharedResource() on huge amount of images > >> > >> The general idea is to mount a single handler, which takes the > >> filename from the url. > >> There is no reason to mount all images by such a handler > one-by-one. > >> > >> Bas > >> > >> Op 10-3-2011 23:01, Ladislav DANKO schreef: > >>> Hi all, > >>> > >>> what is the recommended way to mount huge amount of an images > >>> (thousands) in app? Does mountSharedResource() has any bottleneck? > >>> Simplified code look > >>> like: > >>> > >>> Folder folder = ((Start) Application.get()).getUploadFolder(); > >>> File[] files = folder.getFiles(); > >>> List lList = Arrays.asList(files); > >> Collections.sort(lList); int > >>> i = 0; Iterator iterator = lList.iterator(); > >>> while(iterator.hasNext()) > >>> { > >>> iterator.next(); > >>> String fileName = lList.get(i).getName(); > >>> mountSharedResource("/images/" + fileName, new > >>> ResourceReference(Images.class, fileName).getSharedResourceKey()); > >>> i++; > >>> } > >>> > >>> But what if in folder is for example 100.000 photos? > >>> > >>> Thanks for pointing, > >>> > >>> -- > >>> Ladislav DANKO > >>> > >>> > >>> > >> > - > >>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org > >>> For additional commands, e-mail: users-h...@wicket.apache.org > >>> > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org > > For additional commands, e-mail: users-h...@wicket.apache.org > > > - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org
Re: mountSharedResource() on huge amount of images
Well, exactly. It's usually a debate where both parties have solid arguments and it boils down to preference. I work with both options in medium-sized installations and in the end see the benefits of both options, but prefer disk-based file storage. For most applications though, a CDN or something like Amazon S3 is a great option if you don't want the storage "hassle". I say "hassle" because managing hash-based storage is (imho) easy. It's only a matter of adding a layer of abstraction between file access in the app and actual file storage. Which is something I would always recommend to be able to switch storage if required/desired. This layer of abstraction can then either store files in the db, on disk, both. It can also perform de-duplication, indexing, calculate a hash for hash-based storage, store the file on S3, etc. So to the original author: either way works, pick what you prefer and suits your project best. Bas Op 15-3-2011 19:27, mzem...@osc.state.ny.us schreef: No offense Bas but that seems like a major hassle, especially considering with an ORM its a simple update/delete/find. When it comes time to build out new servers now you have to shuffle around 300k photos instead of simply replicating a database. Also sounds like it would make debugging more difficult when your images are three or more folders deep. And what if you want to store attributes along with the photos? Like say user comments, or flag them for inappropriate content, copyright infringement, etc. What if there is an open file handle when you try to delete the image? Just seems to me a much smoother solution to put in a db. But with that I'm bowing out because as you said it's a heated debate and comes down to personal preference. From: Bas Gooren To: users@wicket.apache.org Date: 03/15/2011 02:10 PM Subject: Re: mountSharedResource() on huge amount of images Now you're talking about rendering them, which is a different topic than mounting a resource which serves said images. You'll only need to mount a single shared resource which serves all the images. However, given the amount of images you can consider allowing your front-end (e.g. apache httd) or a dedicated webserver serve the images. Since you mention that the amount of images can be potentially large (>250), I'd suggest removing the "show all" option, or using an ajax viewport (max 20-50 images on-screen at a time, when the user scrolls you load new images over ajax). Someone else just suggested storing images in the database. While there is usually heated debate about this topic (files on disk vs in the database), let me just say that simply having lots of images is no reason to move images into the database. For starters, you can always store your files in a hashed folder structure, e.g. when the ID is 1234, store the image in a file/folder called /1/2/3/4.jpg Bas Op 15-3-2011 18:00, Ladislav DANKO schreef: Imagine this situation: users have accounts on photo album where they upload images. System from uploaded images create thumbnails. Users can browse their photo - there is combo "show 25", "show 50"... "show all" images. On page I render thumbnails on a page which are shadowbox clickable images. All images ("show all") I do in way described below. Works fine but in extreme situation there is user with more than 3.000 images in one photoalbum. Or -how to do it better way? Thanks, Laco -Original Message- From: Bas Gooren [mailto:b...@iswd.nl] Sent: Thursday, March 10, 2011 11:32 PM To: users@wicket.apache.org Subject: Re: mountSharedResource() on huge amount of images The general idea is to mount a single handler, which takes the filename from the url. There is no reason to mount all images by such a handler one-by-one. Bas Op 10-3-2011 23:01, Ladislav DANKO schreef: Hi all, what is the recommended way to mount huge amount of an images (thousands) in app? Does mountSharedResource() has any bottleneck? Simplified code look like: Folder folder = ((Start) Application.get()).getUploadFolder(); File[] files = folder.getFiles(); ListlList = Arrays.asList(files); Collections.sort(lList); int i = 0; Iteratoriterator = lList.iterator(); while(iterator.hasNext()) { iterator.next(); String fileName = lList.get(i).getName(); mountSharedResource("/images/" + fileName, new ResourceReference(Images.class, fileName).getSharedResourceKey()); i++; } But what if in folder is for example 100.000 photos? Thanks for pointing, -- Ladislav DANKO - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org - To unsubscribe, e
Re: mountSharedResource() on huge amount of images
No offense Bas but that seems like a major hassle, especially considering with an ORM its a simple update/delete/find. When it comes time to build out new servers now you have to shuffle around 300k photos instead of simply replicating a database. Also sounds like it would make debugging more difficult when your images are three or more folders deep. And what if you want to store attributes along with the photos? Like say user comments, or flag them for inappropriate content, copyright infringement, etc. What if there is an open file handle when you try to delete the image? Just seems to me a much smoother solution to put in a db. But with that I'm bowing out because as you said it's a heated debate and comes down to personal preference. From: Bas Gooren To: users@wicket.apache.org Date: 03/15/2011 02:10 PM Subject: Re: mountSharedResource() on huge amount of images Now you're talking about rendering them, which is a different topic than mounting a resource which serves said images. You'll only need to mount a single shared resource which serves all the images. However, given the amount of images you can consider allowing your front-end (e.g. apache httd) or a dedicated webserver serve the images. Since you mention that the amount of images can be potentially large (>250), I'd suggest removing the "show all" option, or using an ajax viewport (max 20-50 images on-screen at a time, when the user scrolls you load new images over ajax). Someone else just suggested storing images in the database. While there is usually heated debate about this topic (files on disk vs in the database), let me just say that simply having lots of images is no reason to move images into the database. For starters, you can always store your files in a hashed folder structure, e.g. when the ID is 1234, store the image in a file/folder called /1/2/3/4.jpg Bas Op 15-3-2011 18:00, Ladislav DANKO schreef: > Imagine this situation: users have accounts on photo album where they upload > images. System from uploaded images create thumbnails. Users can browse > their photo - there is combo "show 25", "show 50"... "show all" images. On > page > I render thumbnails on a page which are shadowbox clickable images. > All images ("show all") I do in way described below. > Works fine but in extreme situation there is user with more than 3.000 > images > in one photoalbum. > > Or -how to do it better way? > > Thanks, > > Laco > > > >> -Original Message- >> From: Bas Gooren [mailto:b...@iswd.nl] >> Sent: Thursday, March 10, 2011 11:32 PM >> To: users@wicket.apache.org >> Subject: Re: mountSharedResource() on huge amount of images >> >> The general idea is to mount a single handler, which takes >> the filename from the url. >> There is no reason to mount all images by such a handler one-by-one. >> >> Bas >> >> Op 10-3-2011 23:01, Ladislav DANKO schreef: >>> Hi all, >>> >>> what is the recommended way to mount huge amount of an images >>> (thousands) in app? Does mountSharedResource() has any bottleneck? >>> Simplified code look >>> like: >>> >>> Folder folder = ((Start) Application.get()).getUploadFolder(); >>> File[] files = folder.getFiles(); >>> List lList = Arrays.asList(files); >> Collections.sort(lList); int >>> i = 0; Iterator iterator = lList.iterator(); >>> while(iterator.hasNext()) >>> { >>> iterator.next(); >>> String fileName = lList.get(i).getName(); >>> mountSharedResource("/images/" + fileName, new >>> ResourceReference(Images.class, fileName).getSharedResourceKey()); >>> i++; >>> } >>> >>> But what if in folder is for example 100.000 photos? >>> >>> Thanks for pointing, >>> >>> -- >>> Ladislav DANKO >>> >>> >>> >> - >>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org >>> For additional commands, e-mail: users-h...@wicket.apache.org >>> > > - > To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org > For additional commands, e-mail: users-h...@wicket.apache.org > Notice: This communication, including any attachments, is intended solely for the use of the individual or entity to which it is addressed. This communication may contain information that is protected from disclosure under State and/or Federal law. Please notify the sender immediately if you have received this communication in error and delete this email from your system. If you are not the intended recipient, you are requested not to disclose, copy, distribute or take any action in reliance on the contents of this information.
Re: mountSharedResource() on huge amount of images
Now you're talking about rendering them, which is a different topic than mounting a resource which serves said images. You'll only need to mount a single shared resource which serves all the images. However, given the amount of images you can consider allowing your front-end (e.g. apache httd) or a dedicated webserver serve the images. Since you mention that the amount of images can be potentially large (>250), I'd suggest removing the "show all" option, or using an ajax viewport (max 20-50 images on-screen at a time, when the user scrolls you load new images over ajax). Someone else just suggested storing images in the database. While there is usually heated debate about this topic (files on disk vs in the database), let me just say that simply having lots of images is no reason to move images into the database. For starters, you can always store your files in a hashed folder structure, e.g. when the ID is 1234, store the image in a file/folder called /1/2/3/4.jpg Bas Op 15-3-2011 18:00, Ladislav DANKO schreef: Imagine this situation: users have accounts on photo album where they upload images. System from uploaded images create thumbnails. Users can browse their photo - there is combo "show 25", "show 50"... "show all" images. On page I render thumbnails on a page which are shadowbox clickable images. All images ("show all") I do in way described below. Works fine but in extreme situation there is user with more than 3.000 images in one photoalbum. Or -how to do it better way? Thanks, Laco -Original Message- From: Bas Gooren [mailto:b...@iswd.nl] Sent: Thursday, March 10, 2011 11:32 PM To: users@wicket.apache.org Subject: Re: mountSharedResource() on huge amount of images The general idea is to mount a single handler, which takes the filename from the url. There is no reason to mount all images by such a handler one-by-one. Bas Op 10-3-2011 23:01, Ladislav DANKO schreef: Hi all, what is the recommended way to mount huge amount of an images (thousands) in app? Does mountSharedResource() has any bottleneck? Simplified code look like: Folder folder = ((Start) Application.get()).getUploadFolder(); File[] files = folder.getFiles(); List lList = Arrays.asList(files); Collections.sort(lList); int i = 0; Iterator iterator = lList.iterator(); while(iterator.hasNext()) { iterator.next(); String fileName = lList.get(i).getName(); mountSharedResource("/images/" + fileName, new ResourceReference(Images.class, fileName).getSharedResourceKey()); i++; } But what if in folder is for example 100.000 photos? Thanks for pointing, -- Ladislav DANKO - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org
RE: mountSharedResource() on huge amount of images
For that many photos I would suggest storing them in a database. Storing that many images on the file system is cumbersome and inefficient in my opinion. You might run into many headaches especially around backups, deploys, upgrades, performance, file names, storage space, etc... From: "Ladislav DANKO" To: Date: 03/15/2011 01:01 PM Subject: RE: mountSharedResource() on huge amount of images Imagine this situation: users have accounts on photo album where they upload images. System from uploaded images create thumbnails. Users can browse their photo - there is combo "show 25", "show 50"... "show all" images. On page I render thumbnails on a page which are shadowbox clickable images. All images ("show all") I do in way described below. Works fine but in extreme situation there is user with more than 3.000 images in one photoalbum. Or -how to do it better way? Thanks, Laco > -Original Message- > From: Bas Gooren [mailto:b...@iswd.nl] > Sent: Thursday, March 10, 2011 11:32 PM > To: users@wicket.apache.org > Subject: Re: mountSharedResource() on huge amount of images > > The general idea is to mount a single handler, which takes > the filename from the url. > There is no reason to mount all images by such a handler one-by-one. > > Bas > > Op 10-3-2011 23:01, Ladislav DANKO schreef: > > Hi all, > > > > what is the recommended way to mount huge amount of an images > > (thousands) in app? Does mountSharedResource() has any bottleneck? > > Simplified code look > > like: > > > > Folder folder = ((Start) Application.get()).getUploadFolder(); > > File[] files = folder.getFiles(); > > List lList = Arrays.asList(files); > Collections.sort(lList); int > > i = 0; Iterator iterator = lList.iterator(); > > while(iterator.hasNext()) > > { > > iterator.next(); > > String fileName = lList.get(i).getName(); > > mountSharedResource("/images/" + fileName, new > > ResourceReference(Images.class, fileName).getSharedResourceKey()); > > i++; > > } > > > > But what if in folder is for example 100.000 photos? > > > > Thanks for pointing, > > > > -- > > Ladislav DANKO > > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org > > For additional commands, e-mail: users-h...@wicket.apache.org > > > - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org Notice: This communication, including any attachments, is intended solely for the use of the individual or entity to which it is addressed. This communication may contain information that is protected from disclosure under State and/or Federal law. Please notify the sender immediately if you have received this communication in error and delete this email from your system. If you are not the intended recipient, you are requested not to disclose, copy, distribute or take any action in reliance on the contents of this information.
RE: mountSharedResource() on huge amount of images
Imagine this situation: users have accounts on photo album where they upload images. System from uploaded images create thumbnails. Users can browse their photo - there is combo "show 25", "show 50"... "show all" images. On page I render thumbnails on a page which are shadowbox clickable images. All images ("show all") I do in way described below. Works fine but in extreme situation there is user with more than 3.000 images in one photoalbum. Or -how to do it better way? Thanks, Laco > -Original Message- > From: Bas Gooren [mailto:b...@iswd.nl] > Sent: Thursday, March 10, 2011 11:32 PM > To: users@wicket.apache.org > Subject: Re: mountSharedResource() on huge amount of images > > The general idea is to mount a single handler, which takes > the filename from the url. > There is no reason to mount all images by such a handler one-by-one. > > Bas > > Op 10-3-2011 23:01, Ladislav DANKO schreef: > > Hi all, > > > > what is the recommended way to mount huge amount of an images > > (thousands) in app? Does mountSharedResource() has any bottleneck? > > Simplified code look > > like: > > > > Folder folder = ((Start) Application.get()).getUploadFolder(); > > File[] files = folder.getFiles(); > > List lList = Arrays.asList(files); > Collections.sort(lList); int > > i = 0; Iterator iterator = lList.iterator(); > > while(iterator.hasNext()) > > { > > iterator.next(); > > String fileName = lList.get(i).getName(); > > mountSharedResource("/images/" + fileName, new > > ResourceReference(Images.class, fileName).getSharedResourceKey()); > > i++; > > } > > > > But what if in folder is for example 100.000 photos? > > > > Thanks for pointing, > > > > -- > > Ladislav DANKO > > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org > > For additional commands, e-mail: users-h...@wicket.apache.org > > > - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org
Re: mountSharedResource() on huge amount of images
The general idea is to mount a single handler, which takes the filename from the url. There is no reason to mount all images by such a handler one-by-one. Bas Op 10-3-2011 23:01, Ladislav DANKO schreef: Hi all, what is the recommended way to mount huge amount of an images (thousands) in app? Does mountSharedResource() has any bottleneck? Simplified code look like: Folder folder = ((Start) Application.get()).getUploadFolder(); File[] files = folder.getFiles(); List lList = Arrays.asList(files); Collections.sort(lList); int i = 0; Iterator iterator = lList.iterator(); while(iterator.hasNext()) { iterator.next(); String fileName = lList.get(i).getName(); mountSharedResource("/images/" + fileName, new ResourceReference(Images.class, fileName).getSharedResourceKey()); i++; } But what if in folder is for example 100.000 photos? Thanks for pointing, -- Ladislav DANKO - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org
mountSharedResource() on huge amount of images
Hi all, what is the recommended way to mount huge amount of an images (thousands) in app? Does mountSharedResource() has any bottleneck? Simplified code look like: Folder folder = ((Start) Application.get()).getUploadFolder(); File[] files = folder.getFiles(); List lList = Arrays.asList(files); Collections.sort(lList); int i = 0; Iterator iterator = lList.iterator(); while(iterator.hasNext()) { iterator.next(); String fileName = lList.get(i).getName(); mountSharedResource("/images/" + fileName, new ResourceReference(Images.class, fileName).getSharedResourceKey()); i++; } But what if in folder is for example 100.000 photos? Thanks for pointing, -- Ladislav DANKO - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org