Wouldn't the cloud / PAAS method actually have you embrace the ephemeral local file storage (assuming it is the most local, most performant, and cheapest file storage). The way I'm doing it is to have a separate app to do index generation, and transfer new indexes to Blob Storage. Each web node can then copy new lucene indexes from Blog storage to their local volatile storage and open the index from there.
This has the following advantages: - It handles n number of instances, since Blob storage is designed to have a high number of instances requesting files at the same time. - Performance is optimised, since each instance is reading lucene indexes locally, independent of each other. The one disadvantage is that the indexing is one-way, in that there is one system or set of systems performing the index generation and pushing, and another set (in the consuming application) only able to read from the index. However, the consuming app servers could add to some kind of a queue which could get picked up by the index generation servers. Pete Peter Miller Head of Technology and Development, Condé Nast Digital -----Original Message----- From: Anders Lybecker [mailto:[email protected]] Sent: 17 January 2011 11:59 To: [email protected] Subject: Re: Using Lucene.Net with Windows Azure Hi Digy, In Azure you have Blob Storage or SQL Azure for persistence, no regular file system. You could use the Azure CloudDrive aka VHD option, which stores the data in Blob Storage anyway. Local file system is not an option, as it is volatile and there is no guarantee that it will be the same between requests. (Just like an Amazon EC2 instance, where the state is lost at reboot.) You got to get your head around the PAAS way of thinking. Evaluate the advantages and disadvantages. The article is about choosing a Lucene storage solution in Azure. I think you have misunderstood Azure, if you think it is all about not making backup. It's still required :-) :-) Anders Lybecker On Sun, Jan 16, 2011 at 9:48 PM, Digy <[email protected]> wrote: > Most probably, I don't understand the problem. > Lucene is intented to be a *fast* search engine on *huge* data and > people are trying to increase the performance using every tricks that > exist. If I don't have enough NW bandwidht comparable to local disk > access why should I use AzureDirectory (or any Webdav-server or iSCSI). > This structure also includes sharing (possible)sensitive data with MS. > Just to avoid backups? > > PS: I would surely give it a try if I could set up a "Azure Blob > Storage - server" on my fast local network. > > DIGY > > > > > -----Original Message----- > From: Anders Lybecker [mailto:[email protected]] > Sent: Sunday, January 16, 2011 9:45 PM > To: [email protected] > Subject: Re: Using Lucene.Net with Windows Azure > > Hi Corey, > > I forgot the link - classic LOL > > Here: > > http://www.lybecker.com/blog/2011/01/16/using-lucene-net-with-microsof > t-azur > e/<http://www.lybecker.com/blog/2011/01/16/using-lucene-net-with-micro > soft-azur%0Ae/> > > :-) > Anders > > On Sun, Jan 16, 2011 at 8:23 PM, Corey Brand <[email protected]> wrote: > > > > > Can you provide a link? > > - Corey > > > > > > > > > Date: Sun, 16 Jan 2011 17:30:31 +0100 > > > Subject: Using Lucene.Net with Windows Azure > > > From: [email protected] > > > To: [email protected] > > > > > > Hi, > > > > > > I have just written an article about using Lucene.Net with > > > Microsoft > > Azure > > > on my blog. > > > > > > I have analyzed the different options, focusing on storage, as it > > > major paradigm shift. > > > > > > Let me know what you think? > > > > > > :-) > > > Anders Lybecker > > > > > > The information contained in this e-mail is of a confidential nature and is intended only for the addressee. If you are not the intended addressee, any disclosure, copying or distribution by you is prohibited and may be unlawful. Disclosure to any party other than the addressee, whether inadvertent or otherwise, is not intended to waive privilege or confidentiality. Internet communications are not secure and therefore Conde Nast does not accept legal responsibility for the contents of this message. Any views or opinions expressed are those of the author. The Conde Nast Publications Ltd (No. 226900), Vogue House, Hanover Square, London W1S 1JU
