If I understand correctly, the issue is that your trident topology queries the same state that’s being updated to compute the result. You can control the number of batches that trident processes simultaneously by adjusting the value of “topology.max.spout.pending”, if you set it to 1 the processing of the batches should be serial. See the pipelining section at http://storm.apache.org/releases/1.0.1/Trident-spouts.html
Thanks, Arun From: Noam Weinberger <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, July 20, 2016 at 2:10 AM To: "[email protected]" <[email protected]> Subject: Trident HBaseState query and update ordering Hello, (I hope I’m sending this to the correct place. If not, I apologize.) I am writing a Trident topology that receives new data from a spout, reads values from HBase, averages the new data with the HBase data, and writes the result to HBase. I initially used the HBaseState library for a separate stateQuery and partitionPersist. However, I’m concerned that it is possible for Batch 2 to query HBase before Batch 1 has updated it (i.e., while Batch 1 is still doing the averaging calculations). In this case, Batch 2 will receive outdated information and write over Batch 1 in HBase. Is there any way to query and write to an external database and enforce ordering through the entire process, such that one batch will not even query until the previous batch has finished updating? Thanks, Noam -------------------------------------------------------- The information transmitted in this email and any of its attachments is intended only for the person or entity to which it is addressed and may contain information concerning Cablevision and/or its affiliates and subsidiaries that is proprietary, privileged, confidential and/or subject to copyright. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited and may be unlawful. If you received this in error, please contact the sender immediately and delete and destroy the communication and all of the attachments you have received and all copies thereof. --------------------------------------------------------
