Hi, Daniel I got the same probem like Sundar. Is that possible to tell me what profiling tool you are using?
thx a lot. /David On Tue, Jul 29, 2008 at 8:19 PM, Daniel Alheiros <[EMAIL PROTECTED]>wrote: > Hi Sundar. > > Well it would be good if you could do some profiling on your Solr app. > I've done it during the indexing process so I could figure out what was > going on in the OutOfMemoryErrors I was getting. > > But you won't definitelly need to have as much memory as your whole > index size. I have a 3.5 million documents (aprox. 10Gb) running on this > 2Gb heap VM. > > Cheers, > Daniel > > -----Original Message----- > From: sundar shankar [mailto:[EMAIL PROTECTED] > Sent: 23 July 2008 23:45 > To: solr-user@lucene.apache.org > Subject: RE: Out of memory on Solr sorting > > > Hi Daniel, > I am afraid that didnt solve my problem. I was guessing my > problem was that I have too much of data and too little memory allocated > for that. I happened to read in couple of the posts which mentioned that > I need VM that is close to the size of my data(folder). I have like 540 > Megs now and a little more than a million and a half docs. Ideally in > that case 512 megs should be enough for me. In fact I am able to perform > all other operations now, commit, optmize, select, update, nightly cron > jobs to index data again. etc etc with no hassles. Even my load tests > perform very well. Just the sort and it doesnt seem to work. I allocated > 2 gigs of memory now. Still same results. Used the GC params u gave me > too. No change what so ever. Am not sure, whats going on. Is there > something that I can do to find out how much is needed in actuality as > my production server might need to be configured in accordance. > > I dont store any documents. We basically fetch standard column data from > oracle database store them into Solr fields. Before I had EdgeNGram > configured and had Solr 1.2, My data size was less that half of what it > is right now. I guess if I remember right, it was of the order of 100 > megs. The max size of a field right now might not cross a 100 chars too. > Quizzled even more now. > > -Sundar > > P.S: My configurations : > Solr 1.3 > Red hat > 540 megs of data (1855013 docs) > 2 gigs of memory installed and allocated like this JAVA_OPTS=$JAVA_OPTS > -Xms2048m -Xmx2048m -XX:MinHeapFreeRatio=50 -XX:NewSize=1024m > -XX:NewRatio=2 -Dsun.rmi.dgc.client.gcInterval=3600000 > -Dsun.rmi.dgc.server.gcInterval=3600000 > > Jboss 4.05 > > > > Subject: RE: Out of memory on Solr sorting > > Date: Wed, 23 Jul 2008 10:49:06 +0100 > > From: [EMAIL PROTECTED] > > To: solr-user@lucene.apache.org > > > > Hi > > > > I haven't read the whole thread so I will take my chances here. > > > > I've been fighting recently to keep my Solr instances stable because > > they were frequently crashing with OutOfMemoryErrors. I'm using Solr > > 1.2 and when it happens there is a bug that makes the index locked > > unless you restart Solr... So in my cenario it was extremelly > damaging. > > > > After some profiling I realized that my major problem was caused by > > the way the JVM heap was being used as I haven't configured it to run > > using any advanced configuration (I had just made it bigger - Xmx and > > Xms 1.5 Gb), it's running on Sun JVM 1.5 (the most recent 1.5 > > available) and it's deployed on a Jboss 4.2 on a RHEL. > > > > So my findings were too many objects were being allocated on the old > > generation area of the heap, which makes them harder to be disposed, > > and also the default behaviour was letting the heap get too filled up > > before kicking a GC and according to the JVM specs the default is if > > after a short period when a full gc is executed if a certain > > percentage of the heap is not freed an OutOfMemoryError should be > thrown. > > > > I've changed my JVM startup params and it's working extremelly stable > > since then: > > > > -Xmx2048m -Xms2048m -XX:MinHeapFreeRatio=50 -XX:NewSize=1024m > > -XX:NewRatio=2 -Dsun.rmi.dgc.client.gcInterval=3600000 > > -Dsun.rmi.dgc.server.gcInterval=3600000 > > > > I hope it helps. > > > > Regards, > > Daniel Alheiros > > > > -----Original Message----- > > From: Fuad Efendi [mailto:[EMAIL PROTECTED] > > Sent: 22 July 2008 23:23 > > To: solr-user@lucene.apache.org > > Subject: RE: Out of memory on Solr sorting > > > > Yes, it is a cache, it stores "sorted" by "sorted field" array of > > Document IDs together with sorted fields; query results can intersect > > with it and reorder accordingly. > > > > But memory requirements should be well documented. > > > > It uses internally WeakHashMap which is not good(!!!) - a lot of > > "underground" warming ups of caches which SOLR is not aware of... > > Could be. > > > > I think Lucene-SOLR developers should join this discussion: > > > > > > /** > > * Expert: The default cache implementation, storing all values in > > memory. > > * A WeakHashMap is used for storage. > > * > > .............. > > > > // inherit javadocs > > public StringIndex getStringIndex(IndexReader reader, String field) > > throws IOException { > > return (StringIndex) stringsIndexCache.get(reader, field); > > } > > > > Cache stringsIndexCache = new Cache() { > > > > protected Object createValue(IndexReader reader, Object fieldKey) > > throws IOException { > > String field = ((String) fieldKey).intern(); > > final int[] retArray = new int[reader.maxDoc()]; > > String[] mterms = new String[reader.maxDoc()+1]; > > TermDocs termDocs = reader.termDocs(); > > TermEnum termEnum = reader.terms (new Term (field, "")); > > .................... > > > > > > > > > > > > Quoting Fuad Efendi <[EMAIL PROTECTED]>: > > > > > I am hoping [new StringIndex (retArray, mterms)] is called only once > > > > per-sort-field and cached somewhere at Lucene; > > > > > > theoretically you need multiply number of documents on size of field > > > > (supposing that field contains unique text); you need not tokenize > > > this field; you need not store TermVector. > > > > > > for 2 000 000 documents with simple untokenized text field such as > > > title of book (256 bytes) you need probably 512 000 000 bytes per > > > Searcher, and as Mark mentioned you should limit number of searchers > > > > in SOLR. > > > > > > So that Xmx512M is definitely not enough even for simple cases... > > > > > > > > > Quoting sundar shankar <[EMAIL PROTECTED]>: > > > > > >> I haven't seen the source code before, But I don't know why the > > >> sorting isn't done after the fetch is done. Wouldn't that make it > > > >> more faster. at least in case of field level sorting? I could be > > > >> wrong too and the implementation might probably be better. But > > >> don't know why all of the fields have had to be loaded. > > >> > > >> > > >> > > >> > > >> > > >>> Date: Tue, 22 Jul 2008 14:26:26 -0700> From: [EMAIL PROTECTED]> To: > > > >>> solr-user@lucene.apache.org> Subject: Re: Out of memory on Solr > > > >>> sorting> > > Ok, after some analysis of FieldCacheImpl:> > - it is > > >>> supposed that (sorted) Enumeration of "terms" is less than > > > >>> total number of documents> (so that SOLR uses specific field type > > > >>> for sorted searches: > solr.StrField with omitNorms="true")> > > > >>> It creates int[reader.maxDoc()] array, checks (sorted) > > >>> Enumeration of > "terms" (untokenized solr.StrField), and > > >>> populates array with document > Ids.> > > - it also creates > > >>> array of String> String[] mterms = new > > >>> String[reader.maxDoc()+1];> > Why > > > > >>> do we need that? For 1G > > >>> document with average term/StrField size > of 100 bytes (which > > >>> could be unique text!!!) it will create kind of > huge 100Gb > > >>> cache > > > > >>> which is not really needed...> StringIndex value = new > > >>> StringIndex > > > > >>> (retArray, mterms);> > If I understand correctly... > > >>> StringIndex _must_ be a file in a > filesystem for such a > case... > > >>> We create StringIndex, and retrieve top > 10 documents, huge > > >>> overhead.> > > > > > >>>> > Quoting Fuad Efendi <[EMAIL PROTECTED]>:> > >> > Ok, what is > > >>> confusing me is implicit guess that FieldCache contains> > "field" > > > >>> and Lucene uses in-memory sort instead of using file-system> > > > > >>> "index".......> >> > Array syze: 100Mb (25M x 4 bytes), and it is > > > >>> just pointers (4-byte> > integers) to documents in index.> >> > > > > >>> org.apache.lucene.search.FieldCacheImpl$10.createValue> > ...> > > > > >>> 357: protected Object createValue(IndexReader reader, Object > > >>> fieldKey)> > 358: throws IOException {> > 359: String field = > > >>> ((String) fieldKey).intern();> > 360: final int[] retArray = new > > > >>> int[reader.maxDoc()]; // > > OutOfMemoryError!!!> > ...> > 408: > > > >>> StringIndex value = new StringIndex (retArray, mterms);> > 409: > > > >>> return value;> > 410: }> > ...> >> > It's very confusing, I don't > > > >>> know such internals...> >> >> >>>>> <field name="XXX" > > >>> type="string" indexed="true" stored="true" > >>>>> > > >>> termVectors="true"/>> >>>>> The sorting is done based on string > > > >>> field.> >> >> > I think Sundar should not use > > >>> [termVectors="true"]...> >> >> >> > Quoting Mark Miller > > >>> <[EMAIL PROTECTED]>:> >> >> Hmmm...I think its 32bits an > > >>> integer with an index entry for each doc, so> >>> >>> >> **25 000 > > > >>> 000 x 32 bits = 95.3674316 megabytes**> >>> >> Then you have the > > > >>> string array that contains each unique term from your> >> > > >>> index...you can guess that based on the number of terms in your > > > >>> index> >> and an avg length guess.> >>> >> There is some other > > >>> overhead beyond the sort cache as well, but thats> >> the bulk of > > > >>> what it will add. I think my memory may be bad with my> >> > > >>> original estimate :)> >>> >> Fuad Efendi wrote:> >>> Thank you > > >>> very much Mark,> >>>> >>> it explains me a lot;> >>>> >>> I am > > >>> guessing: for 1,000,000 documents with a [string] field of > >>> > > > >>> average size 1024 bytes I need 1Gb for single IndexSearcher > >>> > > > >>> instance; field-level cache it is used internally by Lucene (can > > >>> > >>> Lucene manage size if it?); we can't have 1G of such > > >>> documents > >>> without having 1Tb RAM...> >>>> >>>> >>>> >>> > > >>> Quoting Mark Miller <[EMAIL PROTECTED]>:> >>>> >>>> Fuad > > >>> Efendi wrote:> >>>>>> SEVERE: java.lang.OutOfMemoryError: > > >>> allocLargeObjectOrArray - Object> >>>>>> size: 100767936, Num > > >>> elements: 25191979> >>>>>> > > >>>>>>>>> >>>>> I just noticed, this is an exact number of documents > > >>>>>>>>> >>>>> in > > >>> index: 25191979> >>>>>> >>>>> (http://www.tokenizer.org/, you can > > > >>> sort - click headers Id, > >>>>> [COuntry, Site, Price] in a > > >>> table; experimental)> >>>>>> >>>>>> >>>>> If array is allocated > > > >>> ONLY on new searcher warming up I am > >>>>> _extremely_ happy... > > > >>> I had constant OOMs during past month (SUN > >>>>> Java 5).> >>>> > > > >>> It is only on warmup - I believe its lazy loaded, so the first > > > >>> time a> >>>> search is done (solr does the search as part of > > >>> warmup I believe) the> >>>> fieldcache is loaded. The underlying > > > >>> IndexReader is the key to the> > > >>>>>>> fieldcache, so until you get a new IndexReader (SolrSearcher > > >>>>>>> in > > >>> solr> >>>> world?) the field cache will be good. Keep in mind > > >>> that as a searcher> >>>> is warming, the other search is still > > >>> serving, so that will up the ram> >>>> requirements...and since I > > > >>> think you can have >1 searchers on> >>>> deck...you get the > > >>> idea.> >>>>> >>>> As far as the number I gave, thats from a > > >>> memory made months and months> >>>> ago, so go with what you > > >>> see.> >>>>>> >>>>>> >>>>>> > > >>>>>>>> Quoting Fuad Efendi <[EMAIL PROTECTED]>:> >>>>>> >>>>>> I've > > >>>>>>>> even > > >>> seen exceptions (posted here) when "sort"-type queries caused> > > >>>>>>>>> Lucene to allocate 100Mb arrays, here is what happened to > > >>> me:> >>>>>>> >>>>>> SEVERE: java.lang.OutOfMemoryError: > > >>> allocLargeObjectOrArray - Object> >>>>>> size: 100767936, Num > > >>> elements: 25191979> >>>>>> at> >>>>>> > > >>> > > org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl. > > ja > > va:360) > >>>>>> at> >>>>>> > > org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java: > > 72 > > ) - it does not happen after I increased from 4096M to 8192M > >>>>>> > > (JRockit> >>>>>> R27; more intelligent stacktrace, isn't it?)> >>>>>>> > > >>>>>> Thanks Mark; I didn't know that it happens only once (on > > >>>>>> warming > > up a> >>>>>> searcher).> >>>>>>> >>>>>>> >>>>>>> >>>>>> Quoting Mark > > Miller <[EMAIL PROTECTED]>:> >>>>>>> >>>>>>> Because to sort > > efficiently, Solr loads the term to sort on for each> >>>>>>> doc in > > the index into an array. For ints,longs, etc its just an array> > > >>>>>>> the size of the number of docs in your index (i believe > > deleted or> >>>>>>> not). For a String its an array to hold each > > unique string and an array> > > >>>>>>> of ints indexing into the String array.> >>>>>>>> >>>>>>> So > > >>>>>>> if > > you do a sort, and search for something that only gets 1 doc as a> > > >>>>>>> hit...your still loading up that field cache for every single > > doc in> >>>>>>> your index on the first search. With solr, this > > happens in the> >>>>>>> background as it warms up the searcher. The > > end story is, you need more> >>>>>>> RAM to accommodate the sort most > > likely...have you upped your xmx> >>>>>>> setting? I think you can > > roughly say a 2 million doc index would need> >>>>>>> 40-50 MB > > (depending and rough, but to give an idea) per field your> >>>>>>> > > sorting on.> >>>>>>>> >>>>>>> - Mark> >>>>>>>> >>>>>>> sundar shankar > > wrote:> >>>>>>>> Thanks Fuad.> >>>>>>>> But why does just sorting > > provide an OOM. I > >>>>>>>> executed the query without adding the > > sort clause it executed > >>>>>>>> perfectly. In fact I even tried > > remove the maxrows=10 and > >>>>>>>> executed. it came out fine. > > Queries with bigger results > >>>>>>>> seems to come out fine too. But > > > why just sort of that too > >>>>>>>> just 10 rows??> >>>>>>>> -Sundar> > > > >>>>>>>>> > > >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Date: Tue, 22 Jul 2008 > > >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 12:24:35 > > -0700> From: [EMAIL PROTECTED]> > >>>>>>>>> To: > > solr-user@lucene.apache.org> Subject: RE: Out of > >>>>>>>>> memory on > > > Solr sorting> > > >>>>>>>>> > > org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl. > > ja va:403)> > - this piece of code do not request Array[100M] (as I > > seen with > Lucene), it asks only few bytes / Kb for a field...> > > > > Probably > > 128 - 512 is not enough; it is also advisable to use equal sizes> > > -Xms1024M -Xmx1024M> (it minimizes GC frequency, and itensures that > > 1024M is available at startup)> > OOM happens also with fragmented > > memory, when application requests big > contigues fragment and GC is > > unable to optimize; looks like your > application requests a little > > and memory is not available...> > > Quoting sundar shankar > > <[EMAIL PROTECTED]>:> > >> >> >> >> From: > > [EMAIL PROTECTED]> >> To: solr-user@lucene.apache.org> >> > > Subject: Out of memory on Solr sorting> >> Date: Tue, 22 Jul 2008 > > 19:11:02 +0000> >>> >>> >> Hi,> >> Sorry again fellos. I am not sure > > whats happening. The day with > >> solr is bad for me I guess. EZMLM > > didnt let me send any mails this > >> morning. Asked me to confirm > > subscription and when I did, it said I > >> was already a member. Now > > my mails are all coming out bad. Sorry > >> for troubling y'all this > > bad. I hope this mail comes out right.> >> >> > Hi,> > We are > > developing a product in a agile manner and the current > > > > implementation has a data of size just about a 800 megs in dev.> > The > > > memory allocated to solr on dev (Dual core Linux box) is 128-512.> >> > > > My config> > =========> >> > > > <!-- autocommit pending docs if certain criteria are met> > > > <autoCommit>> > <maxDocs>10000</maxDocs>> > <maxTime>1000</maxTime>> > > > > </autoCommit>> > -->> >> > <filterCache> > class="solr.LRUCache"> > > > size="512"> > initialSize="512"> > autowarmCount="256"/>> >> > > > <queryResultCache> > class="solr.LRUCache"> > size="512"> > > > initialSize="512"> > autowarmCount="256"/>> >> > <documentCache> > > > class="solr.LRUCache"> > size="512"> > initialSize="512"> > > > autowarmCount="0"/>> >> > > > <enableLazyFieldLoading>true</enableLazyFieldLoading>> >> >> > My > > Field> > > > =======> >> > <fieldType name="autocomplete" > > > class="solr.TextField">> <analyzer type="index">> > <tokenizer > > class="solr.KeywordTokenizerFactory"/>> > <filter > > class="solr.LowerCaseFilterFactory" />> > <filter > > class="solr.PatternReplaceFilterFactory" > > pattern="([^a-z0-9])" > > replacement="" replace="all" />> > <filter > > class="solr.EdgeNGramFilterFactory" > > maxGramSize="100" > > minGramSize="1" />> > </analyzer>> > <analyzer type="query">> > > > <tokenizer class="solr.KeywordTokenizerFactory"/>> > <filter > > class="solr.LowerCaseFilterFactory" />> > <filter > > class="solr.PatternReplaceFilterFactory" > > pattern="([^a-z0-9])" > > replacement="" replace="all" />> > <filter > > class="solr.PatternReplaceFilterFactory" > > pattern="^(.{20})(.*)?" > > replacement="$1" replace="all" />> > </analyzer>> > </fieldType>> >> > > >> > > > Problem> > ======> >> > I execute a query that returns 24 rows of > > result. I pick 10 out of > > it. I have no problem when I execute > > this.> > > > But When I do sort it by a String field that is fetched from this > > > > > > > result. I get an OOM. I am able to execute several> > other queries > > with no problem. Just having a sort asc clause added > > to the query > > throws an OOM. Why is that.> > What should I have ideally done. My > > config on QA is pretty similar > > to the dev box and probably has > > more data than on dev.> > It didnt throw any OOM during the > > integration test. The Autocomplete > > is a new field we added > > recently.> >> > Another point is that the indexing is done with a > > field of type string> > <field name="XXX" type="string" indexed="true" > > > stored="true" > > termVectors="true"/>> >> > and the autocomplete > > field is a copy field.> > > >> > The sorting is done based on string field.> >> > Please do lemme > > know what mistake am I doing?> >> > Regards> > Sundar> >> > P.S: The > > stack trace of the exception is> >> >> > Caused by: > > org.apache.solr.client.solrj.SolrServerException: Error > > executing > > query> > at > > > > org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest > > .j > > ava:86)> > at > > > > org.apache.solr.client.solrj.impl.BaseSolrServer.query(BaseSolrServer. > > ja > > va:101)> > at > > > > com.apollo.sisaw.solr.service.AbstractSolrSearchService.makeSolrQuery( > > Ab stractSolrSearchService.java:193)> > ... 105 more> > Caused by: > > org.apache.solr.common.SolrException: Java heap space > > > > java.lang.OutOfMemoryError: Java heap space> > at > > > > org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl. > > ja > > va:403)> > at > > org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java: > > 72 > > )> > at > > > > org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl. > > ja > > va:352)> > at > > > > org.apache.lucene.search.FieldSortedHitQueue.comparatorString(FieldSor > > te > > dHitQueue.java:416)> > at > > > > org.apache.lucene.search.FieldSortedHitQueue$1.createValue(FieldSorted > > Hi > > tQueue.java:207)> > at > > org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java: > > 72 > > )> > at > > > > org.apache.lucene.search.FieldSortedHitQueue.getCachedComparator(Field > > So > > rtedHitQueue.java:168)> > at > > > > > org.apache.lucene.search.FieldSortedHitQueue.<init>(FieldSortedHitQueue. > > java:56)> > at > > > > > org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher. > > java:907)> > at > > > > org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher > > .j > > ava:838)> > at > > > > org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java > > :2 > > 69)> > at > > > > > org.apache.solr.handler.component.QueryComponent.process(QueryComponent. > > java:160)> > at > > > > org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sear > > ch > > Handler.java:156)> > at > > > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle > > rB > > ase.java:128)> > at > > org.apache.solr.core.SolrCore.execute(SolrCore.java:1025)> > at > > > > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter. > > ja > > va:338)> > at > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter > > .j > > ava:272)> > at > > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > > ca > > tionFilterChain.java:202)> > at > > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi > > lt > > erChain.java:173)> > at > > > > org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFil > > te > > r.java:96)> > at > > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > > ca > > tionFilterChain.java:202)> > at > > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi > > lt > > erChain.java:173)> > at > > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa > > lv > > e.java:213)> > at > > > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa > > lv > > e.java:178)> > at > > > > org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(Security > > As > > sociationValve.java:175)> > at > > > > org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve > > .j > > ava:74)> > at > > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja > > va > > :126)> > at > > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja > > va > > :105)> > at > > > > org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnec > > ti > > onValve.java:156)> > at > > > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve. > > java:107)> > at > > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java > > :1 > > 48)> > at > > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java: > > 86 > > 9)> >> > > > _________________________________________________________________> > > > Wish to Marry Now? Click Here to Register FREE> > > >>>>>>>>> > > http://www.shaadi.com/registration/user/index.php?ptnr=mhottag>> > > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> > > _________________________________________________________________> > > >>>>>>>> Missed your favourite programme? Stop surfing TV channels and > > > >>>>>>>> > start planning your weekend TV viewing with our > >>>>>>>> > > comprehensive TV Listing> >>>>>>>> > > http://entertainment.in.msn.com/TV/TVListing.aspx> >>>>>>>>> >>>>>> > > >>>>>> >>>>>> >>>> >>>> >>>> > > >>> > > > >>>> > > >> _________________________________________________________________ > > >> Wish to Marry Now? Join Shaadi.com FREE! > > >> http://www.shaadi.com/registration/user/index.php?ptnr=mhottag > > > > > > > > > > http://www.bbc.co.uk/ > > This e-mail (and any attachments) is confidential and may contain > personal views which are not the views of the BBC unless specifically > stated. > > If you have received it in error, please delete it from your system. > > Do not use, copy or disclose the information in any way nor act in > reliance on it and notify the sender immediately. > > Please note that the BBC monitors e-mails sent or received. > > Further communication will signify your consent to this. > > > > _________________________________________________________________ > Missed your favourite programme? Stop surfing TV channels and start > planning your weekend TV viewing with our comprehensive TV Listing > http://entertainment.in.msn.com/TV/TVListing.aspx > > http://www.bbc.co.uk/ > This e-mail (and any attachments) is confidential and may contain personal > views which are not the views of the BBC unless specifically stated. > If you have received it in error, please delete it from your system. > Do not use, copy or disclose the information in any way nor act in reliance > on it and notify the sender immediately. > Please note that the BBC monitors e-mails sent or received. > Further communication will signify your consent to this. > >
