We have a requirement as follows:
Large streams of Time-ordered data received form over a TCP source to be 
processed in a time-ordered (cannot do parallel processing of the records) 
The raw data or processed data does not need to be retained/cached in the 
Ignite layer.
The processed records need to be inserted into a DB.

The SLA is 
a) Lowest Latency b/w the stream ingestion till DB. 
b) Reliability : cannot afford losing any record/packet from the stream

In view of the above SLAs, would appreciate any feedback/suggestions on 
the following :

Q1) what would be a better choice for Stream Ingestion from TCP source : 
Ignite SocketStreamer OR Kafka Streamer ? 

Q2) since the records are time-ordered and need to be processed in the 
same manner,  we need a FIFO Queue in which the Streamer can inject the 
raw records, which can be picked up by a ClusterSingletonService which can 
process the raw records in a time-ordered manner & then insert the 
processed record in a write-through Cache to DB. 
Pls note that to ensure Reliability , the take() of each record from the 
FIFO Queue by the ClusterSingletonService till it writes the processed 
record to the Cache --> needs to be completed within single User 
Transaction, such that in the event of any failure/node-crash/etc while 
processing the record, it must rollback the transaction and NOT remove the 
raw record from the FIFO Queue.
I tried IgniteQueue, but it does not support User Transaction. 
What would be the best alternative for the FIFO Queue implementation ?

Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

Reply via email to