Hi Peter, That is exactly what AzureDirectory does, but at a segment level. When you write to the index, the segment is pushed to Blob Storage. When reopening an IndexReader the Blob Storage is queried to see if new segments are added. If there is they will automatically be downloaded to ephemeral local file storage. When searching the ephemeral local file storage is used.
:-) Anders Lybecker On Mon, Jan 17, 2011 at 4:45 PM, Peter Miller <[email protected]>wrote: > Wouldn't the cloud / PAAS method actually have you embrace the ephemeral > local file storage (assuming it is the most local, most performant, and > cheapest file storage). The way I'm doing it is to have a separate app to do > index generation, and transfer new indexes to Blob Storage. Each web node > can then copy new lucene indexes from Blog storage to their local volatile > storage and open the index from there. > > This has the following advantages: > - It handles n number of instances, since Blob storage is designed to have > a high number of instances requesting files at the same time. > - Performance is optimised, since each instance is reading lucene indexes > locally, independent of each other. > > The one disadvantage is that the indexing is one-way, in that there is one > system or set of systems performing the index generation and pushing, and > another set (in the consuming application) only able to read from the index. > However, the consuming app servers could add to some kind of a queue which > could get picked up by the index generation servers. > > Pete > > Peter Miller > Head of Technology and Development, Condé Nast Digital > > > -----Original Message----- > From: Anders Lybecker [mailto:[email protected]] > Sent: 17 January 2011 11:59 > To: [email protected] > Subject: Re: Using Lucene.Net with Windows Azure > > Hi Digy, > > In Azure you have Blob Storage or SQL Azure for persistence, no regular > file system. > You could use the Azure CloudDrive aka VHD option, which stores the data in > Blob Storage anyway. > > Local file system is not an option, as it is volatile and there is no > guarantee that it will be the same between requests. (Just like an Amazon > EC2 instance, where the state is lost at reboot.) You got to get your head > around the PAAS way of thinking. Evaluate the advantages and disadvantages. > The article is about choosing a Lucene storage solution in Azure. > > I think you have misunderstood Azure, if you think it is all about not > making backup. It's still required :-) > > :-) > Anders Lybecker > > On Sun, Jan 16, 2011 at 9:48 PM, Digy <[email protected]> wrote: > > > Most probably, I don't understand the problem. > > Lucene is intented to be a *fast* search engine on *huge* data and > > people are trying to increase the performance using every tricks that > > exist. If I don't have enough NW bandwidht comparable to local disk > > access why should I use AzureDirectory (or any Webdav-server or iSCSI). > > This structure also includes sharing (possible)sensitive data with MS. > > Just to avoid backups? > > > > PS: I would surely give it a try if I could set up a "Azure Blob > > Storage - server" on my fast local network. > > > > DIGY > > > > > > > > > > -----Original Message----- > > From: Anders Lybecker [mailto:[email protected]] > > Sent: Sunday, January 16, 2011 9:45 PM > > To: [email protected] > > Subject: Re: Using Lucene.Net with Windows Azure > > > > Hi Corey, > > > > I forgot the link - classic LOL > > > > Here: > > > > http://www.lybecker.com/blog/2011/01/16/using-lucene-net-with-microsof > > t-azur > > e/<http://www.lybecker.com/blog/2011/01/16/using-lucene-net-with-micro > > soft-azur%0Ae/> > > > > :-) > > Anders > > > > On Sun, Jan 16, 2011 at 8:23 PM, Corey Brand <[email protected]> wrote: > > > > > > > > Can you provide a link? > > > - Corey > > > > > > > > > > > > > Date: Sun, 16 Jan 2011 17:30:31 +0100 > > > > Subject: Using Lucene.Net with Windows Azure > > > > From: [email protected] > > > > To: [email protected] > > > > > > > > Hi, > > > > > > > > I have just written an article about using Lucene.Net with > > > > Microsoft > > > Azure > > > > on my blog. > > > > > > > > I have analyzed the different options, focusing on storage, as it > > > > major paradigm shift. > > > > > > > > Let me know what you think? > > > > > > > > :-) > > > > Anders Lybecker > > > > > > > > > > > > The information contained in this e-mail is of a confidential nature and is > intended only for the addressee. If you are not the intended addressee, any > disclosure, copying or distribution by you is prohibited and may be > unlawful. Disclosure to any party other than the addressee, whether > inadvertent or otherwise, is not intended to waive privilege or > confidentiality. Internet communications are not secure and therefore Conde > Nast does not accept legal responsibility for the contents of this message. > Any views or opinions expressed are those of the author. > > The Conde Nast Publications Ltd (No. 226900), Vogue House, Hanover Square, > London W1S 1JU >
