Hi, Ignite can go so far in meeting your requirements for durable storage and retrieval, but a primary node failure is probably going to be terminal because unless you can boot a machine in less than 5 seconds you can forget about reloading ignite anyway, you simply won't be able to restore your TCP connection in time. (Unless of course you have some radical kind of hardware with OS in bubble memory or similar, but even then?)
So you need to use something like a p2p protocol such as bit torrent where the download is broken into chunks and delegated to multiple nodes each holding some or all of the blobs data. The client has to be able to recover as well, either by round robin or by implementing a p2p protocol - you still can't depend on a single fallible point of entry to your service because of the problem above. If you have such a p2p protocol I'm not sure Ignite adds anything much of value? John -----Original Message----- From: steven <[email protected]> Sent: Thursday, May 23, 2019 2:16 AM To: [email protected] Subject: Is Apache Ignite appropriate for my use case? Email received from outside the company. If in doubt don't click links nor open attachments! ________________________________ Hi, I need to manage a large fleet of servers (around 100k machines). Each node contains a subset of the data (basically binary blobs) in memory. The data store is a separate database. I am trying to provide a binary blob lookup service (in greatly simplified terms). For each incoming request (which contains a binary string), every node should perform matching of the string against its subset of the data. When the first matching binary blob is found, that result is returned, and all other nodes should stop searching. If no matches are found, NOT_FOUND should be returned. Requests must be handled within 5 seconds. If any node fails, it must be revived with the same subset of data that it had before. One challenge is how to handle node failure in the middle of a request. It is unlikely that a node will be revived quickly enough to respond within 5 seconds. Most likely there should be standby nodes that will retrieve the failing node's subset of data from the data store and perform matching upon being notified of node failure. Is Apache Ignite appropriate for this use case? Thanks, Steven -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
