[CONF] Apache Samza > SEP-8: Add in-memory system consumer & producer

[Bharath Kumarasubramanian (Confluence)]

Title: Message Title

Bharath Kumarasubramanian edited a page

SEP-8: Add in-memory system consumer & producer ... Approach A - Use existing BlockingEnvelopeMap and have one common class that shares the responsibility of consumer as well as producer. The class will be responsible for handling both producing and consuming messages off the same queue. Approach B - Have separate producer and consumer. Tie up the consumer with the producer so that producer has hooks to produce to the same underlying `BlockingEnvelopeMap` that consumer uses. Approach C - Have separate consumer and producer. Introduce a custom queue that are shared between consumer and producer. The queue lifecycle is managed by the SystemAdmin. I am leaning towards Approach C as its simpler, not tied to BlockingEnvelopeMap and has separation of concerns. Test Plan High level stateless application ... Low level application with custom IME   Code Block language java theme Emacs collapse true /** * Sample test case using collection based system for low level application using custom IME. * It demonstrates the use of IME as a data source as opposed to raw message. The users are responsible for creating a * complete IME object with partition information. */   ... ...   ImmutableList<MyIME> inputData = Utils.createMyIME(...);   List<OV> outputData = ... // mutable   StreamDescriptor.Input<Object, Object> input = StreamDescriptor.<>input("input-stream-low-level-app") .from(inputData); StreamDescriptor.Output<OK,OV> output = StreamDescriptor.<OK,OV>output("output-stream-low-level-app") .from(outputData);   // application logic StreamTaskApplication app = StreamTaskApplication.create(config, new MyTaskFactory()) .addInputs(Collections.singletonList(input, changelog)) .addOutputs(Collections.singletonList(output)); app.run(); app.waitForFinish(); // assertions on outputData ... View page • Add comment • Like

Stop watching space • Manage notifications This message was sent by Atlassian Confluence 5.8.17