Hi Mason, While evaluating both streaming platforms, we had setup a 3-node staging cluster of Linux nodes (each with 40 cores and 128 GB RAM).
We essentially tried out implementing the following 3 broad set of functionalities on both platforms: 1) *Intra stream/Per Message* processing (e.g. filter/transform an input message to an output message based on some logic), 2) *Intra stream/Across Message* aggregations (e.g. group by certain messages based on a particular key and aggregate (e.g. sum) some other fields that act as measures), 3) *Inter-stream/Across Message* processing (e.g. joining 2 streams on a particular key). The source/target systems used by our evaluation topologies were a combination of proprietary streaming systems, internally deployed high throughput/low latency K-V stores, as well as other open-source systems like Kafka. The core idea of our evaluation was to simply play around and see whether each platform is functionally rich enough to support a broad set of application use-cases across the company, and at the same time provide the necessary robustness/fault-tolerance. It is quite evident by the exhaustive set of evaluation criteria as outlined in the 3rd blog <http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3>. We didn't focus on the performance numbers of any platform, since we considered the stability/operational aspects to be of far more importance. Regards, Satish On Tue, Oct 20, 2015 at 6:15 PM, Mason Yu <[email protected]> wrote: > Gentlemen: > > I would interested as far as the type of streaming for the > data ingestion used for both types of Big Data platforms, I would > also be interested in the size and topology of the Linux clusters and > the sizing of the nodes. > Please advise. > > Best, > > Mason Yu Jr. > Big Data Architect > > > 著名的孫子 > > On Mon, Oct 19, 2015 at 7:24 AM, Satish Mittal <[email protected]> > wrote: > >> [image: Boxbe] <https://www.boxbe.com/overview> This message is eligible >> for Automatic Cleanup! ([email protected]) Add cleanup rule >> <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Ftoken%3DwI09zqfTG1J%252F3GblUrzW0Pqc76yAHki1V3%252BlFhYDazcfoEYfNXfKYIWrgbfJFBUvBT0TCO6sp5uErcrt7wu9Ks09ejDxgipbgVcQWrodMDikSrsOn2UAUDp84jGflbp6MqB8nnTj7bBApqJeuwQx0A%253D%253D%26key%3Dxe1S0TwIhB2NQA2X78iQpX%252BLBukrIwhTv5Yzebm974E%253D&tc_serial=23014816890&tc_rand=1230885861&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> >> | More info >> <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=23014816890&tc_rand=1230885861&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> >> >> Hi All, >> >> The data platform team at Inmobi recently performed an extensive >> evaluation exercise in the process of finalizing the real-time Stream >> processing stack as the choice of our platform. >> >> We have captured all the details of our evaluation as the following >> series of 4 blogs which have been published at Inmobi technology site: >> >> 1) Introduction; Identifying stream processing use-cases at Inmobi; >> Identifying potential Technology Candidates. In the interest of time, we >> limited the >> >> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1 >> >> 2) Detailed overview of Storm and Spark Streaming platforms >> >> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2 >> >> 3) Identify and define various important evaluation criteria >> >> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3 >> >> 4) Detailed findings on various evaluation criteria, evaluation summary >> along with the final recommendation. >> >> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4 >> >> We hope that this analysis would be useful in general to anyone who is >> starting to explore the world of real-time stream processing and decide >> upon a particular tech stack. >> >> Please go through the blogs and let us know your thoughts! >> >> Regards, >> Satish >> >> >> >> >> >> _____________________________________________________________ >> The information contained in this communication is intended solely for >> the use of the individual or entity to whom it is addressed and others >> authorized to receive it. It may contain confidential or legally privileged >> information. If you are not the intended recipient you are hereby notified >> that any disclosure, copying, distribution or taking any action in reliance >> on the contents of this information is strictly prohibited and may be >> unlawful. If you have received this communication in error, please notify >> us immediately by responding to this email and then delete it from your >> system. The firm is neither liable for the proper and complete transmission >> of the information contained in this communication nor for any delay in its >> receipt. >> > > -- _____________________________________________________________ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
