Hey guys, Sorry for the delay. I somehow read the replies but forgot to reply :)
I think the target of Infispector and Zipkin based tracing would be different, although there is some common ground. I think both Infispector and Zipkin would be helpful in helping diagnose cluster performance issues and get a general overview of how messages pass from one node to the other. In terms of difference Infispector seems more geared towards education/qe whereas Zipkin is more targeted as something we can run at runtime in production for our users. Although both projects have different targets I think we'll be able to take advantage of each others work, e.g. custom JGroups protocols points, byteman/bytebuddy rules for intercept scenarions...etc. Great work leading all the Infispector work Tomas, looking forward to demos/videos :D Cheers, -- Galder Zamarreño Infinispan, Red Hat > On 10 May 2016, at 15:53, Radim Vansa <rva...@redhat.com> wrote: > > To complement this, MFT is a tool that won't offer any sleek charts or > visualisations. It's tricky to use and understand - it's intended for > developers as a tool for problem analysis. But it gets more in depth > than InfiSpector, linking the information from different nodes, JFR > events and so forth. > > R. > > On 05/10/2016 09:41 AM, Tomas Sykora wrote: >> Hello Galder, and all! >> It’s nice to communicate again via infinispan-dev after a while :) >> >> TL;DR: I can see some intersections with zipkin.io initiative goals but >> InfiSpector seems to be much more “easier to handle and contribute to at >> this moment” -- that suits more our student-related use case. Let’s continue >> with the discussion :) >> >> Firstly, a short introduction into the context. Red Hat is running Research >> & Development laboratory here in Brno at 2 biggest local universities: >> Masaryk University, Faculty of Informatics (FI MU) and Brno University of >> Technology, Faculty of Information Technologies (FIT VUT). >> The aim is to better and sooner reach out to students, get them involved >> into interesting project, show them open-source, git, git workflows and many >> other things (project specific). An year ago I got excited about this idea >> and started to think whether I can deliver such a project. And I did. >> >> Team faces one big challenge and this is a time constraint. Students are >> working on _several_ projects during their studies to fulfill courses’ >> requirements to pass the semester. It’s hard for them to find additional >> time to be coding even something else. Team managed that but slowly, that’s >> understandable though. Designing InfiSpector infrastructure took us some >> time (Kafka, Druid, NodeJS) + evaluation of these technologies + proof of >> concepts. >> >> All 5 team members are 2nd year students of bachelor studies at FIT VUT Brno. >> Marek Ciz (https://github.com/mciz), also my very good friend from my home >> town :) His primary domain is Druid, Kafka and infrastructure. >> Vratislav Hais (https://github.com/vratislavhais) Primary domain is >> front-end. >> Jan Fitz (https://github.com/janfitz) Primary domain is front-end and >> graphic design (also designed our logo). >> Tomas Veskrna -- starting >> Patrik Cigas -- starting >> >> At this moment we are very close to getting real data to be monitored via >> web UI. It’s a matter of 1-2 months considering there is an examination >> period happening now at the University. >> >> ******************* >> What is InfiSpector and what we want to achieve: >> >> * We missed graphical representation of Infinispan nodes communication so we >> want >> -- To be able to spot possible issues AT THE FIRST LOOK (e.g. wait, this >> should be coordinator, how is that possible he sends/receives only 10 % of >> all messages?) >> -- To demonstrate nicely what’s happening inside of ISPN cluster for >> newcomers (to see how Infinispan nodes talk to each other to better >> understand concepts) >> -- To be using nice communication diagrams that describes flows like (130 >> messages from node1 to node5 -- click to see them in detail, filter out in >> more detail) >> * We aimed for NON-invasive solution >> -- No changes in Infinispan internal code >> -- Just add custom JGroups protocol, gather data and send them where you >> want [0] >> * Provide historical recording of JGroups communication >> * Help to analyze communication recording from big data point of view >> -- No need to manually go through gigabytes of text trace logs >> >> Simplified InfiSpector architecture: >> >> Infinispan Cluster (JGroups with our protocol) ---> Apache Kafka ---> Druid >> Database (using Kafka Firehose to inject Kafka Topic) <---> NodeJS server >> back-end <---> front-end (AngularJS) >> >> What is coming out from custom JGroup protocol is a short JSON document [1] >> with a timestamp, sending and receiving node, length of a message and the >> message itself. Other stuff can be added easily. >> >> We will be able to easily answer queries like: >> How many messages were sent from node1 to node3 during “last” 60 seconds? >> What are these messages? >> How many of them were PutKeyValueCommands? >> Filter out Heart beats (or even ignore them completely), etc. >> >> We don’t have any video recording yet but we are very close to that point. >> From UI perspective we will be using these 2 charts: [2], [3]. >> >> >> Talking about Infinispan 9 plans -- [4] was reported a month ago by you >> Galder and we are working on InfiSpector actively let’s say 5 months -- it >> looks like I should have advertised InfiSpector more, sooner, but I was >> waiting for at least first working demo to start with blogging and videos :) >> It’s good that you’ve noticed and that we are having this conversation right >> now. >> >> To be honest I find http://zipkin.io/ initiative to be quite similar. >> However, InfiSpector seems to be much more “easier” and not targeting at >> performance analysis directly. Just adding one protocol at protocol stack >> and you are good to go. We were thinking about putting Kafka and Druid >> somewhere into the cloud (later) so users don’t need to start all that big >> infrastructure at their local machines. >> >> I am very open to anything that will help us as a community to achieve our >> common goal -- to be able to graphically monitor Infinispan communication. >> Additionally I would be _personally_ looking for something that is easily >> achievable and is suitable for students to quickly learn new things and >> quickly make meaningful contributions. >> >> Thanks! >> Tomas >> >> [0] Achieved by custom JGroups protocol -- JGROUPS_TO_KAFKA protocol has >> been implemented. This can be added at the end of JGroups stack and every >> single message that goes through that is sent to Apache Kafka. >> [1] >> { >> "direction": "receiving/up", >> "src": "tsykora-19569", >> "dest": "tsykora-27916", >> "length": 182, >> "timestamp": 1460302055376, >> "message": "SingleRpcCommand{cacheName='___defaultcache', >> command=PutKeyValueCommand{key=f6d52117-8a27-475e-86a7-002a54324615, >> value=tsykora-19569, flags=null, putIfAbsent=false, >> valueMatcher=MATCH_ALWAYS, >> metadata=EmbeddedExpirableMetadata{lifespan=60000, maxIdle=-1, >> version=null}, successful=true}}" >> } >> [2] http://bl.ocks.org/NPashaP/9796212 >> [3] http://bl.ocks.org/mbostock/1046712 >> [4] https://issues.jboss.org/browse/ISPN-6346 >> >> >> >> >> ----- Original Message ----- >>> From: "Galder Zamarreño" <gal...@redhat.com> >>> To: "infinispan -Dev List" <infinispan-dev@lists.jboss.org>, "Tomas Sykora" >>> <tsyk...@redhat.com> >>> Sent: Monday, May 9, 2016 5:06:06 PM >>> Subject: Infispector >>> >>> Hi all, >>> >>> I've just noticed [1], @Thomas, it appears this is your baby? Could you >>> explain in more detail what you are trying to achieve with this? Do you have >>> a video to show what exactly it does? >>> >>> Also, who's [2]? Curious to know who's working on this stuff :) >>> >>> The reason I'm interested in finding out a bit more about [1] is because we >>> have several efforts in the distributed monitoring/tracing area and want to >>> make sure we're not duplicating same effort. >>> >>> 1. Radim's Message Flow Tracer [3]: This is a project to tool for tracing >>> messages and control flow in JGroups/Infinispan using Byteman. >>> >>> 2. Zipkin effort [4]: The idea behind is to have a way to have Infinispan >>> cluster-wide tracing that uses Zipkin to capture and visualize where time is >>> spent within Infinispan. This includes both JGroups and other components >>> that could be time consuming, e.g. persistence. This will be main task for >>> Infinispan 9. This effort will use a lot of interception points Radim has >>> developed in [3] to tie together messages related to a request/tx around the >>> cluster. >>> >>> Does your effort fall within any of these spaces? >>> >>> Cheers, >>> >>> [1] https://github.com/infinispan/infispector >>> [2] https://github.com/mciz >>> [3] https://github.com/rvansa/message-flow-tracer >>> [4] https://issues.jboss.org/browse/ISPN-6346 >>> -- >>> Galder Zamarreño >>> Infinispan, Red Hat >>> >>> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Radim Vansa <rva...@redhat.com> > JBoss Performance Team > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
