If you keep copy=false (default), loop sends the same exchange for each iteration. This allows you to manipulate headers inside the loop and subsequent iterations would see those header changes.
On Wed, Jan 31, 2024 at 2:18 AM Ghassen Kahri <ghassen.ka...@codeonce.fr> wrote: > Hi Jeremy, > > The idea of using the loop EIP crossed my mind as well, but I'm uncertain > about the feasibility of manipulating headers for each iteration. > > I appreciate your concern. > > Thank you. > > Le lun. 29 janv. 2024 à 18:35, Jeremy Ross <jeremy.g.r...@gmail.com> a > écrit : > > > > To achieve this, I iterated through the route X times, each time > > executing > > a query with a different offset. I utilized Camel headers to store the > > offset and other flags, as mentioned in my initial email. > > > > This is a perfectly reasonable approach IMO. > > > > > Does Camel have any built-in functionality that > > accomplishes the same task? Additionally, since I was "improvising," I'm > > curious if my code adheres to best practices. I sensed that it might not, > > given that I implemented business logic at the route level. > > > > The EIPs are the building blocks that allow you to accomplish this type > of > > use case. Apart from EIPs, Camel doesn't have specific functionality to > > query and process paged resources. The Loop EIP ( > > https://camel.apache.org/components/4.0.x/eips/loop-eip.html) might be a > > little more idiomatic than a route calling itself recursively. > > > > > > On Fri, Jan 26, 2024 at 3:07 AM Ghassen Kahri <ghassen.ka...@codeonce.fr > > > > wrote: > > > > > Hey Raymond, I appreciate your response. > > > > > > We are both on board with the idea of dividing the query response into > > > chunks. Let's discuss the "how" in Camel. > > > > > > To achieve this, I iterated through the route X times, each time > > executing > > > a query with a different offset. I utilized Camel headers to store the > > > offset and other flags, as mentioned in my initial email. > > > > > > My primary question is: Does Camel have any built-in functionality that > > > accomplishes the same task? Additionally, since I was "improvising," > I'm > > > curious if my code adheres to best practices. I sensed that it might > not, > > > given that I implemented business logic at the route level. > > > > > > Le jeu. 25 janv. 2024 à 15:46, ski n <raymondmees...@gmail.com> a > écrit > > : > > > > > > > Yes, dividing it into chunks is a good practice. This adheres to > > > > message-based systems in general, not specific to Camel. > > > > Let's discuss both ways of processing messages: > > > > > > > > 1. One big message > > > > > > > > Say the message is 100 GB+ and this is processed by some integration > > > > software on a server, you need to scale the server > > > > for that amount. This means both memory and CPU must be capable of > > doing > > > > processing so amount of data. When you want to perform > > > > EIP's (like filters or transformation) this will be difficult, > because > > > the > > > > needed resources to match that. > > > > > > > > Say this big message comes one's a week, then you have a very big > > server > > > > basically run for nothing. > > > > > > > > 2. Many small messages > > > > > > > > Because of 1 it's generally the best practice to have fixed sized > > smaller > > > > messages. When possible, directly on the source. > > > > If this is somehow not possible, you can split them and move it back > > to a > > > > Kafka topic, then you use streaming the messages > > > > and do the actual EIP's on the small message. Some advantages are: > > > > > > > > 1. Predictable: Every message is of the same size, so you load test > > this > > > > and match resources. > > > > 2. Resources: A small message needs less resources (CPU/Memory) to > > > process > > > > 3. Load: The load is spread over time (you can use a smaller server). > > > > 4. Realtime: You don't need to wait until all data is gathered and > then > > > > send it in batch, but > > > > you can process it when it happens. > > > > 5. Scaling: When the load is high, you may add multiple threads or > even > > > > multiple pods/containers to scale, when you > > > > don't need it anymore, you can scale back. > > > > > > > > Raymond > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 25, 2024 at 2:32 PM Ghassen Kahri < > > ghassen.ka...@codeonce.fr > > > > > > > > wrote: > > > > > > > > > Hello community, > > > > > > > > > > I am currently working on a feature within the Camel project that > > > > involves > > > > > processing Kafka messages (String) and performing a query based on > > that > > > > > message. Initially, I implemented a classic route that called a > > service > > > > > method responsible for executing the query. However, I encountered > an > > > > issue > > > > > with the size of the query result, as the memory couldn't handle > > such a > > > > > massive amount of data. > > > > > > > > > > In response to this challenge, I devised an alternative solution > that > > > > might > > > > > be considered unconventional. The approach involves querying the > > > database > > > > > multiple times and retrieving the results in manageable chunks. > > > > > Consequently, the route needs to be executed multiple times. The > > > current > > > > > structure of my route is as follows: > > > > > > > > > > > > > > > from(getInput()) > > > > > .routeId(getRouteId()) > > > > > > > > > > .bean(Service.class, "extractDataInChunks") > > > > > > > > > > .choice() > > > > > > > > > > .when(header(PAGINATION_END_FLAG).isEqualTo(true)).to(getOutput()) > > > > > > > > > > > > > > > > > > > > > > > > > .when(header(PAGINATION_END_FLAG).isEqualTo(false)).to(getOutput(),directUri(getRouteId())); > > > > > //re-execute the route with offset = offset+limit > > > > > > > > > > > > > > > The extractDataInChunks method queries the database with a > > > parameterized > > > > > limit (chunk size) and an offset that ranges from 0 to X * limit. > The > > > > > PAGINATION_END_FLAG is a Camel header, initially set to false, and > is > > > > > switched to true by the extractDataInChunks method if the size of > the > > > > query > > > > > result is 0. > > > > > > > > > > I would appreciate feedback on whether this solution adheres to > good > > > > Camel > > > > > practices, specifically the consideration of implementing business > > > logic > > > > at > > > > > the route level. Additionally, I am curious if there are any > built-in > > > > > Enterprise Integration Patterns (EIPs) in Camel that might be more > > > > suitable > > > > > for my business requirements. > > > > > > > > > > Thank you for your insights. > > > > > > > > > > > > > > >
