Mark, I changed the code to use HTTP 1.1 in InvokeHTTP, but it did not help.
I looked into "Shutdown" part of this link https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/#okhttpclient <https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/#okhttpclient>. After implementing this, the processor didn't hang. This defeats the purpose of the thread pool and connection reuse, but I was able to get past the issue. It is still a good idea to get https://issues.apache.org/jira/browse/NIFI-8181 <https://issues.apache.org/jira/browse/NIFI-8181> implemented. Regards, Vijay > On Jan 29, 2021, at 1:34 PM, Vijay Chhipa <[email protected]> wrote: > > Joe, > > behavior is the same with the 1.12.1 version. > > Mark, > I will make the change to the OkHttp client and try. I do know that this flow > was working for a long time and there was possibly an upgrade done on the > source endpoint. Thank you for opening the Jira. > > Regards, > Vijay > >> On Jan 28, 2021, at 4:37 PM, Mark Payne <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hey Vijay, >> >> I’ve seen a few people lately running into issues with InvokeHTTP. The >> common thread for all of them is that they are hitting servers that are >> using HTTP 2. Reading threads from OkHttp (the underlying HTTP library that >> we use), I see that a lot of people are running into issues with it. In at >> least several of the cases, it ends up being the HTTP Server or a proxy/load >> balancer in the middle that is not properly supporting the HTTP 2 protocol. >> Unclear if there are HTTP 2.0 related bugs in OkHttp itself or not. In any >> case, users often comment that changing the protocol to HTTP 1.1 resolved >> the issue. I filed a Jira [1] to allow that to be exposed in the InvokeHTTP >> processor. Hopefully that will help. >> >> Thanks >> -Mark >> >> [1] https://issues.apache.org/jira/browse/NIFI-8181 >> <https://issues.apache.org/jira/browse/NIFI-8181> >> >>> On Jan 28, 2021, at 4:46 PM, Joe Witt <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> The likely relevant thread is here >>> >>> "Timer-Driven Process Thread-9" #70 prio=5 os_prio=31 cpu=12025.30ms >>> elapsed=4403.88s tid=0x00007fe44f16b000 nid=0xe103 in Object.wait() >>> [0x000070000fed4000] >>> java.lang.Thread.State: WAITING (on object monitor) >>> at java.lang.Object.wait([email protected]/Native Method) >>> - waiting on <no object reference available> >>> at java.lang.Object.wait([email protected]/Object.java:328) >>> at okhttp3.internal.http2.Http2Stream.waitForIo(Http2Stream.java:577) >>> at >>> okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:143) >>> - waiting to re-lock in wait() <0x00000007e007d818> (a >>> okhttp3.internal.http2.Http2Stream) >>> at >>> okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120) >>> at >>> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >>> at >>> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) >>> at >>> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) >>> at >>> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >>> at >>> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) >>> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) >>> at okhttp3.RealCall.execute(RealCall.java:69) >>> at >>> org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:793) >>> at >>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >>> at >>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) >>> at >>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) >>> at >>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) >>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:515) >>> at >>> java.util.concurrent.FutureTask.runAndReset([email protected]/FutureTask.java:305) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run([email protected]/ScheduledThreadPoolExecutor.java:305) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628) >>> at java.lang.Thread.run([email protected]/Thread.java:834) >>> >>> This implies that it is either genuinely waiting for IO that the remote >>> server is failing to send or has gotten itself into a state it could never >>> break out of. Can you please use this exact same flow on the latest NiFi >>> release 1.12.1 to see if the issue remains? >>> >>> Thanks >>> >>> On Thu, Jan 28, 2021 at 2:43 PM Vijay Chhipa <[email protected] >>> <mailto:[email protected]>> wrote: >>> Hi Joe, >>> >>> Thanks for looking into this issue. >>> We are on NiFi 1.10.0 >>> >>> Please see the attached >>> 1. thread dump file >>> 2. nifi-app.log >>> 3. nifi-bootstrap.log >>> 4. nifi.properties >>> >>> The InvokeHTTP configuration is as shown below. >>> >>> >>> >>> >>> >>>> On Jan 25, 2021, at 7:24 AM, Joe Witt <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> Hello >>>> >>>> If you suspect an actual stuck/hung thread then the best course of action >>>> is to generate a thread dump. This can be done in a couple ways but one >>>> of the easiest is to run 'bin/nifi.sh dump'. We would need to see the >>>> nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should >>>> include the specific nifi version being used specifics of the processor >>>> configuration. >>>> >>>> Thanks >>>> >>>> On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> Hello, >>>> >>>> In my case something similar happens. I invoke a REST-JSON service that >>>> sometimes takes more than 10 minutes to respond (I have set the timeout to >>>> 15 minutes) and after the second or third call the service responds (I >>>> could check it on the server side) but the processor remains waiting for >>>> the response. I try to stop it and after "terminate" it but the thread >>>> seems to stay active, so I delete it, (disconnect it and remove it from >>>> the flow) and add a new InvokeHttp and it works again. >>>> >>>> Given the difficulty of finding steps to reproduce it, I have not reported >>>> it. I hope someone can help us. >>>> >>>> On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> Hi All, >>>> >>>> I am seeing a case where InvokeHTTP processor hangs after several >>>> successful calls to an endpoint. (Processor is paginating over millions of >>>> rows and pulling a few thousand rows at a time) >>>> >>>> >>>> <Screen Shot 2021-01-24 at 10.37.14 PM.png> >>>> >>>> If I have more than one thread scheduled, all threads hang. >>>> <Screen Shot 2021-01-24 at 10.37.55 PM.png> >>>> >>>> >>>> When this happens InvokeHTTP does not release the current flowfile at all. >>>> My only recourse is to try to stop the processor. >>>> <Screen Shot 2021-01-17 at 9.36.08 PM.png> >>>> >>>> This "stop" does not complete, at which point I have to terminate the >>>> processor. >>>> <Screen Shot 2021-01-17 at 9.36.58 PM.png> >>>> >>>> After termination of this processor, starting of the processor does not >>>> process any files. So I have to remove the processor and all its >>>> connections, delete the processor and replace it with another InvokeHTTP >>>> with same properties and connections. >>>> >>>> I have tried to set the "Connection: close" header but it didn't help. >>>> >>>> Has anyone seen this behaviors, how did you resolve it? >>>> >>>> Thanks >>>> >>>> >>>> >>>> -- >>>> Jairo Henao >>>> @jairohenaorojas >>>> >>> >> >
smime.p7s
Description: S/MIME cryptographic signature
