Let me be more detailed in my response: Kafka works on “at least once” semantics. Therefore, given your assumption that Kafka "will be operational", we can assume that at least once semantics will hold.
At this point, it comes down to designing for consumer (really Spark Executor) resilience. From a DC standpoint, you can use an in memory data fabric, like is provided by InsightEdge, http://insightedge.io/docs/010/index.html <http://insightedge.io/docs/010/index.html>. In this case, WAN replication out to other DCs is available at the Data Grid layer. See here: http://www.gigaspaces.com/Data-Replication <http://www.gigaspaces.com/Data-Replication>. Assuming that the consumers respect at least once semantics (that is: don’t attempt to keep track of the offset or any other state), then Spark can parallelize execution using Dumb Consumers. The backing data fabric can do what it does best, which is conflict resolution in the case that a DC goes down for a period of time. One advantage of this architecture is that it can be used to load balance, reducing infrastructure costs. Of course, the CAP theorem is still in play, so things like intra-DC latencies and consistency SLAs need to be considered. But in principle, you can balance competing concerns against one another based on business requirements. HTH > On Apr 19, 2016, at 10:53 AM, Erwan ALLAIN <eallain.po...@gmail.com> wrote: > > Cody, you're right that was an example. Target architecture would be 3 DCs :) > Good point on ZK, I'll have to check that. > > About Spark, both instances will run at the same time but on different > topics. That would be quite useless to have to 2DCs working on the same set > of data. > I just want, in case of crash, that the healthy spark works on all topics > (retrieve dead spark load). > > Does it seem an awkward design ? > > On Tue, Apr 19, 2016 at 5:38 PM, Cody Koeninger <c...@koeninger.org > <mailto:c...@koeninger.org>> wrote: > Maybe I'm missing something, but I don't see how you get a quorum in only 2 > datacenters (without splitbrain problem, etc). I also don't know how well ZK > will work cross-datacenter. > > As far as the spark side of things goes, if it's idempotent, why not just run > both instances all the time. > > > > On Tue, Apr 19, 2016 at 10:21 AM, Erwan ALLAIN <eallain.po...@gmail.com > <mailto:eallain.po...@gmail.com>> wrote: > I'm describing a disaster recovery but it can be used to make one datacenter > offline for upgrade for instance. > > From my point of view when DC2 crashes: > > On Kafka side: > - kafka cluster will lose one or more broker (partition leader and replica) > - partition leader lost will be reelected in the remaining healthy DC > > => if the number of in-sync replicas are above the minimum threshold, kafka > should be operational > > On downstream datastore side (say Cassandra for instance): > - deploy accross the 2 DCs in (QUORUM / QUORUM) > - idempotent write > > => it should be ok (depends on replication factor) > > On Spark: > - treatment should be idempotent, it will allow us to restart from the last > commited offset > > I understand that starting up a post crash job would work. > > Question is: how can we detect when DC2 crashes to start a new job ? > > dynamic topic partition (at each kafkaRDD creation for instance) + topic > subscription may be the answer ? > > I appreciate your effort. > > On Tue, Apr 19, 2016 at 4:25 PM, Jason Nerothin <jasonnerot...@gmail.com > <mailto:jasonnerot...@gmail.com>> wrote: > It the main concern uptime or disaster recovery? > >> On Apr 19, 2016, at 9:12 AM, Cody Koeninger <c...@koeninger.org >> <mailto:c...@koeninger.org>> wrote: >> >> I think the bigger question is what happens to Kafka and your downstream >> data store when DC2 crashes. >> >> From a Spark point of view, starting up a post-crash job in a new data >> center isn't really different from starting up a post-crash job in the >> original data center. >> >> On Tue, Apr 19, 2016 at 3:32 AM, Erwan ALLAIN <eallain.po...@gmail.com >> <mailto:eallain.po...@gmail.com>> wrote: >> Thanks Jason and Cody. I'll try to explain a bit better the Multi DC case. >> >> As I mentionned before, I'm planning to use one kafka cluster and 2 or more >> spark cluster distinct. >> >> Let's say we have the following DCs configuration in a nominal case. >> Kafka partitions are consumed uniformly by the 2 datacenters. >> >> DataCenter Spark Kafka Consumer Group Kafka partition (P1 to P4) >> DC 1 Master 1.1 >> >> Worker 1.1 my_group P1 >> Worker 1.2 my_group P2 >> DC 2 Master 2.1 >> >> Worker 2.1 my_group P3 >> Worker 2.2 my_group P4 >> >> I would like, in case of DC crash, a rebalancing of partition on the healthy >> DC, something as follow >> >> DataCenter Spark Kafka Consumer Group Kafka partition (P1 to P4) >> DC 1 Master 1.1 >> >> Worker 1.1 my_group P1, P3 >> Worker 1.2 my_group P2, P4 >> DC 2 Master 2.1 >> >> Worker 2.1 my_group P3 >> Worker 2.2 my_group P4 >> >> I would like to know if it's possible: >> - using consumer group ? >> - using direct approach ? I prefer this one as I don't want to activate WAL. >> >> Hope the explanation is better ! >> >> >> On Mon, Apr 18, 2016 at 6:50 PM, Cody Koeninger <c...@koeninger.org >> <mailto:c...@koeninger.org>> wrote: >> The current direct stream only handles exactly the partitions >> specified at startup. You'd have to restart the job if you changed >> partitions. >> >> https://issues.apache.org/jira/browse/SPARK-12177 >> <https://issues.apache.org/jira/browse/SPARK-12177> has the ongoing work >> towards using the kafka 0.10 consumer, which would allow for dynamic >> topicparittions >> >> Regarding your multi-DC questions, I'm not really clear on what you're >> saying. >> >> On Mon, Apr 18, 2016 at 11:18 AM, Erwan ALLAIN <eallain.po...@gmail.com >> <mailto:eallain.po...@gmail.com>> wrote: >> > Hello, >> > >> > I'm currently designing a solution where 2 distinct clusters Spark (2 >> > datacenters) share the same Kafka (Kafka rack aware or manual broker >> > repartition). >> > The aims are >> > - preventing DC crash: using kafka resiliency and consumer group mechanism >> > (or else ?) >> > - keeping consistent offset among replica (vs mirror maker,which does not >> > keep offset) >> > >> > I have several questions >> > >> > 1) Dynamic repartition (one or 2 DC) >> > >> > I'm using KafkaDirectStream which map one partition kafka with one spark. >> > Is >> > it possible to handle new or removed partition ? >> > In the compute method, it looks like we are always using the currentOffset >> > map to query the next batch and therefore it's always the same number of >> > partition ? Can we request metadata at each batch ? >> > >> > 2) Multi DC Spark >> > >> > Using Direct approach, a way to achieve this would be >> > - to "assign" (kafka 0.9 term) all topics to the 2 sparks >> > - only one is reading the partition (Check every x interval, "lock" stored >> > in cassandra for instance) >> > >> > => not sure if it works just an idea >> > >> > Using Consumer Group >> > - CommitOffset manually at the end of the batch >> > >> > => Does spark handle partition rebalancing ? >> > >> > I'd appreciate any ideas ! Let me know if it's not clear. >> > >> > Erwan >> > >> > >> >> > > > >
