I'd be happy to provide more information as you request. I just wasn't aware of which metrics might be important factors. Anyway, it sounds like you feel this goes beyond people suggesting from their experience or providing other related ideas.
Of course I did already test around with the tools I mentioned below, and others too. From what I could read Kafka is not designed for that large data. In various discussions it was suggested to use an object storage for the large data and use Kafka for messaging referencing the object keys. This is the solution I'm currently testing with and feel this is the most reliable. I'll do further tests. Thanks. On 24.03.23 15:30, Jeremy McMillan wrote:
That's a big question, and it isn't clear whether there's a large or small ratio of reads to writes between these microservices, for example. It isn't clear what your latency tolerance is for these large transfers either. This sounds like a big endeavor, and if there's money to be made, your best bet is to get architecture advice with an NDA so that the architecture can take all of the cost/risk/benefit factors into consideration. Asking free software community to give you blind architecture advice will not get you much closer to a decision than you already are. If it's not constrained by deadlines and r&d budget, maybe your best bet is to try a couple of things out and compare what you can squeeze out of each? Maybe you want to experiment before you choose an architectural design? We would still need to understand more of your performance goals to help design an experiment. It's hard to be the first or only person with a good idea. The difference between what you're imagining and what you can see already extant in the world might be constrained by execution and not imagination. Please consider sharing more info. FWIW, that's the other side of the old "bike shed" parable if you are seeking input from others. On Fri, Mar 24, 2023, 06:55 Thomas Kramer <[email protected]> wrote: Hi all, do you have any feedback on this? Or is this rather a question for StackOverflow? Thanks. On 21.03.23 10:00, Thomas Kramer wrote: > I'm considering Ignite for a ondemand-scalable microservice-oriented > architecture. I'd use the memory cache for shared data across the > microservices. Maybe I would also use Ignite compute for distributed > tasks, however I believe the MOA philosophy would recommend REST for > this. > > My question is rather about large data transfer between the > microservices. In addition to smaller amount of data shared in the > caches across all microservices, I need to constantly send large data > blocks (50M-500M) between the microservices, typically from one sender > to one receiver. There is no need to persist these on disk. > > Would Ignite be fast and efficient for this? What size of chunks should > the data be split? Or would I better use Kafka in parallel to Ignite to > transfer the large data blocks? Or maybe go even more low-level with > something like ZeroMQ? > > Thanks for comments and suggestions. > >
