Re: [DISCUSS] Splitting up the Indexing Topology

2017-09-27 Thread Michael Miklavcic
+1 to the split. I also feel it's much easier to dissect problems when these actions are separated. It's also easier to fine tune each independently, which may have additional performance benefits. M On Mon, Sep 25, 2017 at 5:31 PM, James Sirota wrote: > I have experienced issues with ES and HD

Re: [DISCUSS] Splitting up the Indexing Topology

2017-09-25 Thread James Sirota
I have experienced issues with ES and HDFS indexing in production and have previously split out the topologies into two separate topologies. As you state the benefits of this approach are (a) tuning each topology separately, (b) ability to attribute problems to a specific topology (why is somet

[DISCUSS] Splitting up the Indexing Topology

2017-09-25 Thread Casey Stella
One of the lessons that have bubbled up in doing some performance analysis is that having the indexing topology share both the ES and the HDFS writer in the same topology can be problematic from a tuning perspective. Specifically, it's hard to square that circle and make both perform fast enough to