I don't know if actual stack frames are used up, but each iteration of the loop definitely shows up in the route history. For this reason, I don't use loop for iterating through a large number of items.
On Wed, Jan 31, 2024 at 12:31 PM Anthony Wu <anthony...@toasttab.com> wrote: > Hi folks - I had thought that the loop EIP was meant only for testing > purposes? In the 3.14.x LTS docs the doc page reads, my emphasis: > > The Loop EIP allows for processing a message a number of times, possibly in > a different way for each iteration. _Useful mostly during testing._ > > See > > https://stackoverflow.com/questions/51257248/camel-stackoverflow-error-when-route-is-called-recursively > as well. > > In the past I've used a SEDA queue like the following in Java DSL: > > > from("seda:foo").process(processorThatTerminatesWhenBodyIsExhausted).to("seda:foo") > > Any insight on whether the loop EIP is safe to use (no longer suffers from > memory overrun) here is greatly appreciated. > > On Wed, Jan 31, 2024 at 8:45 AM Jeremy Ross <jeremy.g.r...@gmail.com> > wrote: > > > If you keep copy=false (default), loop sends the same exchange for each > > iteration. This allows you to manipulate headers inside the loop and > > subsequent iterations would see those header changes. > > > > On Wed, Jan 31, 2024 at 2:18 AM Ghassen Kahri <ghassen.ka...@codeonce.fr > > > > wrote: > > > > > Hi Jeremy, > > > > > > The idea of using the loop EIP crossed my mind as well, but I'm > uncertain > > > about the feasibility of manipulating headers for each iteration. > > > > > > I appreciate your concern. > > > > > > Thank you. > > > > > > Le lun. 29 janv. 2024 à 18:35, Jeremy Ross <jeremy.g.r...@gmail.com> a > > > écrit : > > > > > > > > To achieve this, I iterated through the route X times, each time > > > > executing > > > > a query with a different offset. I utilized Camel headers to store > the > > > > offset and other flags, as mentioned in my initial email. > > > > > > > > This is a perfectly reasonable approach IMO. > > > > > > > > > Does Camel have any built-in functionality that > > > > accomplishes the same task? Additionally, since I was "improvising," > > I'm > > > > curious if my code adheres to best practices. I sensed that it might > > not, > > > > given that I implemented business logic at the route level. > > > > > > > > The EIPs are the building blocks that allow you to accomplish this > type > > > of > > > > use case. Apart from EIPs, Camel doesn't have specific functionality > to > > > > query and process paged resources. The Loop EIP ( > > > > https://camel.apache.org/components/4.0.x/eips/loop-eip.html) might > > be a > > > > little more idiomatic than a route calling itself recursively. > > > > > > > > > > > > On Fri, Jan 26, 2024 at 3:07 AM Ghassen Kahri < > > ghassen.ka...@codeonce.fr > > > > > > > > wrote: > > > > > > > > > Hey Raymond, I appreciate your response. > > > > > > > > > > We are both on board with the idea of dividing the query response > > into > > > > > chunks. Let's discuss the "how" in Camel. > > > > > > > > > > To achieve this, I iterated through the route X times, each time > > > > executing > > > > > a query with a different offset. I utilized Camel headers to store > > the > > > > > offset and other flags, as mentioned in my initial email. > > > > > > > > > > My primary question is: Does Camel have any built-in functionality > > that > > > > > accomplishes the same task? Additionally, since I was > "improvising," > > > I'm > > > > > curious if my code adheres to best practices. I sensed that it > might > > > not, > > > > > given that I implemented business logic at the route level. > > > > > > > > > > Le jeu. 25 janv. 2024 à 15:46, ski n <raymondmees...@gmail.com> a > > > écrit > > > > : > > > > > > > > > > > Yes, dividing it into chunks is a good practice. This adheres to > > > > > > message-based systems in general, not specific to Camel. > > > > > > Let's discuss both ways of processing messages: > > > > > > > > > > > > 1. One big message > > > > > > > > > > > > Say the message is 100 GB+ and this is processed by some > > integration > > > > > > software on a server, you need to scale the server > > > > > > for that amount. This means both memory and CPU must be capable > of > > > > doing > > > > > > processing so amount of data. When you want to perform > > > > > > EIP's (like filters or transformation) this will be difficult, > > > because > > > > > the > > > > > > needed resources to match that. > > > > > > > > > > > > Say this big message comes one's a week, then you have a very big > > > > server > > > > > > basically run for nothing. > > > > > > > > > > > > 2. Many small messages > > > > > > > > > > > > Because of 1 it's generally the best practice to have fixed sized > > > > smaller > > > > > > messages. When possible, directly on the source. > > > > > > If this is somehow not possible, you can split them and move it > > back > > > > to a > > > > > > Kafka topic, then you use streaming the messages > > > > > > and do the actual EIP's on the small message. Some advantages > are: > > > > > > > > > > > > 1. Predictable: Every message is of the same size, so you load > test > > > > this > > > > > > and match resources. > > > > > > 2. Resources: A small message needs less resources (CPU/Memory) > to > > > > > process > > > > > > 3. Load: The load is spread over time (you can use a smaller > > server). > > > > > > 4. Realtime: You don't need to wait until all data is gathered > and > > > then > > > > > > send it in batch, but > > > > > > you can process it when it happens. > > > > > > 5. Scaling: When the load is high, you may add multiple threads > or > > > even > > > > > > multiple pods/containers to scale, when you > > > > > > don't need it anymore, you can scale back. > > > > > > > > > > > > Raymond > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 25, 2024 at 2:32 PM Ghassen Kahri < > > > > ghassen.ka...@codeonce.fr > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hello community, > > > > > > > > > > > > > > I am currently working on a feature within the Camel project > that > > > > > > involves > > > > > > > processing Kafka messages (String) and performing a query based > > on > > > > that > > > > > > > message. Initially, I implemented a classic route that called a > > > > service > > > > > > > method responsible for executing the query. However, I > > encountered > > > an > > > > > > issue > > > > > > > with the size of the query result, as the memory couldn't > handle > > > > such a > > > > > > > massive amount of data. > > > > > > > > > > > > > > In response to this challenge, I devised an alternative > solution > > > that > > > > > > might > > > > > > > be considered unconventional. The approach involves querying > the > > > > > database > > > > > > > multiple times and retrieving the results in manageable chunks. > > > > > > > Consequently, the route needs to be executed multiple times. > The > > > > > current > > > > > > > structure of my route is as follows: > > > > > > > > > > > > > > > > > > > > > from(getInput()) > > > > > > > .routeId(getRouteId()) > > > > > > > > > > > > > > .bean(Service.class, "extractDataInChunks") > > > > > > > > > > > > > > .choice() > > > > > > > > > > > > > > > > .when(header(PAGINATION_END_FLAG).isEqualTo(true)).to(getOutput()) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > .when(header(PAGINATION_END_FLAG).isEqualTo(false)).to(getOutput(),directUri(getRouteId())); > > > > > > > //re-execute the route with offset = offset+limit > > > > > > > > > > > > > > > > > > > > > The extractDataInChunks method queries the database with a > > > > > parameterized > > > > > > > limit (chunk size) and an offset that ranges from 0 to X * > limit. > > > The > > > > > > > PAGINATION_END_FLAG is a Camel header, initially set to false, > > and > > > is > > > > > > > switched to true by the extractDataInChunks method if the size > of > > > the > > > > > > query > > > > > > > result is 0. > > > > > > > > > > > > > > I would appreciate feedback on whether this solution adheres to > > > good > > > > > > Camel > > > > > > > practices, specifically the consideration of implementing > > business > > > > > logic > > > > > > at > > > > > > > the route level. Additionally, I am curious if there are any > > > built-in > > > > > > > Enterprise Integration Patterns (EIPs) in Camel that might be > > more > > > > > > suitable > > > > > > > for my business requirements. > > > > > > > > > > > > > > Thank you for your insights. > > > > > > > > > > > > > > > > > > > > > > > > > > > >
