Samza looks really good. Many concepts are exactly what is needed. But we have some specific requirements and I am not sure how to implement them with Samza.
Background --------------- We have components which corresponds to Samza Job/Task. Some are universal components – for example FileReader, JMSListener, SCPSender, PDFCreator, etc. Such reusable components are used in many different business processes. And we have business process specific components – fore example InvoiceGenerator, ClientsImporter, ProductListFilter, etc... Components can be either message sources or transformers. Message source components don't have inputs, only output. Transformers have inputs and outputs. We have a notation of business process – very similar to BPMN. Business definition has name and flow/topology description and also every component in business process can have process specific configuration. For example: process InvoiceExport, might have flow: InvoiceGenerator -> PDFCreator -> SCPSender And process OverduePaymentExport, flow: OverduePaymentGenerator -> PDFCreator -> SCPSender Both processes use same components PDFCreator and SCPSender but components configurations may differ. Questions ------------ As I understand Samza doesn't have something like Business Process. We like to run different Business Process at the same time with Samza which are isolated form each other. So do we must use different input stream names to isolate processes data flows form each other? For example process InvoiceExport use streams named: InvoiceExport.PDFCreator.stream and process OverduePaymentExport use streams named: OverduePaymentExport.PDFCreator.stream And also it seems that we can't start Business Process as a whole, but instead start components on by one. And process shutting down means shutting down components also one by one. Any ideas how this can be automated? Currently our business process configuration file includes configuration for all components which are used in process. For example different business processes have different queue name for JMSListener component. I am confused how to translate this to Samza job configurations. Create some sort of custom translator which generates Samza job configurations? We don't have actually a lot of messages and initially we don't need speedy processing. But not losing a single message and reliability is needed. What is the current state of Samza? Can we use it in production? Can you give same time hint when stable release is out? Please can anyone help me? Thank you for your time toivo
