The main issue we are trying to work around is the 2 messages that are taking off the input queue from the pipeline. I stated that When I was using 2.6 there were 2 consumers, 1 for meta and 1 consumer. This version only took 1 message off the queue As expected.. 2.10.x has 3 consumers 1 meta and Im assuming 2 consumers for messages.. I observe it taking 2 messages off the queue. SIGTERM then causes the pipe to shutdown and the second message is lost..
This happens with and without kubernetes.. I just observed it with the move to kubernetes and shutdowns are more prevalent. Daniel Cosio dcco...@gmail.com > On May 29, 2024, at 2:51 AM, Pablo Duboue <pablo.dub...@gmail.com> wrote: > > On Tue, May 28, 2024 at 3:40 PM Daniel Cosio <dcco...@gmail.com > <mailto:dcco...@gmail.com>> wrote: > >> I’m seeing a couple of issues with the pipelines that cause messages(and >> CAS’s) to be lost. >> >> The pipeline is prefetching 2 messages off the input queue. If the >> pipeline receives a SIGTERM is will shut down. In this case, the document >> has finished but we loose the second document that was prefetched.. >> I never see the second prefetched document back on the input queue, so Im >> assuming it must have be ack’d(which doesn’t seem that this should >> happen)..I also seem to loose both CAS’s from the reader because >> The callback to the reader to tell it the CAS can be released is never >> called… >> >> >> This issue surfaced when moving our implementation to Kubernetes.. The >> shutdown is happening when a pod is stopped. I have a process that will >> requeue the last DOCID which is ok, but I have no way of recovering the >> second prefetched document.. >> >> I use UimaJ and Uima-AS 2.10.2. We cannot upgrade to 3.x yet. >> >> When I was running on 2.6 there were 2 consumers connected to the input >> queue.. With 2.10.2 there are 3 and Im suspecting the extra consumer is the >> one grabbing the second message. >> >> So, a few questions. >> >> 1. Is there a way via config to stop the pipeline from prefetching the >> second document >> 2. Is there a way to (via a shutdown hook) to let the pipeline return >> CAS’s to the reader if it recieves a SIGTERM. >> 3. Is there a way go back to the 2 consumers? >> >> Any insight would be appreciated >> > > Hi Daniel, > > What you describe is very unfortunate. > > I did a search on the mailing lists and found this issue that talks about > Kubernets and SIGTERM changes to the shutdown hooks: > https://issues.apache.org/jira/browse/UIMA-5909 > > You might want to look at what was done to close it and try reverting that > change to see if it fixes your issue. > > The three consumers are explained in this email: > https://lists.apache.org/thread/sm3g4yszsnygj3gxsxj16b6q6nozwcxb > > It doesn't seem the jump from 2 consumers to 3 might be related to what > you're observing. > > Hope this helps, > > P > > > > > > > > >> >> >> Daniel Cosio >> dcco...@gmail.com <mailto:dcco...@gmail.com>
