On Thu, Jun 18, 2015 at 8:55 AM, Ravikumar Govindarajan < [email protected]> wrote:
> Apologize for resurrecting this thread… > > One problem of lucene is OS buffer-cache pollution during segment merges, > as documented here > > http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html > > This problem could occur in Blur, when short-circuit reads are enabled... > True but Blur deals with this issue by not allowing (by default) the merges to effect the Block Cache. > > My take on this… > > It may be possible to overcome the problem by simply re-directing > merge-read requests to a node other than local-node instead of fancy stuff > like O_DIRECT, FADVISE etc... > I have always thought of having merge occur in a Mapreduce (or Yarn) job instead of locally. > > In a mixed setup, this means merge requests need to be diverted to low-end > Rack2 machines {running only data-nodes} while short-circuit read requests > will continue to be served from high-end Rack1 machines {running both > shard-server and data-nodes} > > Hadoop 2.x provides a cool read-API "seekToNewSource" > API documentation says "Seek to given position on a node other than the > current node" > From blur code, it's just enough if we open a new FSDataInputStream for > merge-reads and issue seekToNewSource call. Once merges are done, it can > closed & discarded… > > Please let know your view-points on this… > We could do this, but I find that reading the TIM file types over the wire during a merge causes a HUGE slow down in merge performance. The fastest way to merge is to copy the TIM files involved in the merge locally to run the merge and then delete them after the fact. Aaron > > -- > Ravi > > On Mon, Mar 9, 2015 at 5:45 PM, Ravikumar Govindarajan < > [email protected]> wrote: > > > > > On Sat, Mar 7, 2015 at 11:00 AM, Aaron McCurry <[email protected]> > wrote: > > > >> > >> I thought the normal hdfs replica rules were once local. One remote rack > >> once same rack. > >> > > > > Yes. One copy is local & other two copies on the same remote rack. > > > > How did > >> land on your current configuration ? > > > > > > When I was evaluating disk-budget, we were looking at 6 expensive drives > > per machine. It lead me to think what those 6 drives would do & how we > can > > reduce the cost. Then stumbled on this two-rack setup and now we need > only > > 2 such drives... > > > > Apart from reduced disk-budget & write-overhead on cluster, it also helps > > in greater availability as rack-failure would be recoverable... > > > > -- > > Ravi > > > > >
