Hi Joe, I can reproduce the error. I have a three node secured NiFi 1.14. I've tested with DetectDuplicate and it only worked with db index 0 in RedisConnectionPoolService. If any other index, failed with message:
DetectDuplicate[id=c4a64927-6056-1d2d-ffff-ffffa3ac489e] Failed to process session due to All sentinels down, cannot determine where is clnifi master is running...; Processor Administratively Yielded for 1 sec: redis.clients.jedis.exceptions.JedisConnectionException: All sentinels down, cannot determine where is clnifi master is running... This is a 3 node Redis/Sentinel without unsecure (without SSL). Eduardo Fontes On Fri, Aug 6, 2021 at 8:09 AM Joe Gresock <jgres...@gmail.com> wrote: > Thanks for the attached flow, Jens. So just to confirm, if you use > Database Index 0, it works on both 1.13.2 and 1.14.0, but if you switch > Database Index to 3, it only works on 1.13.2? > > I was not able to reproduce this in Standalone mode, but I have not tried > Sentinel mode. Can you verify whether Standalone mode works for you? > > It's still puzzling that there would be SSL exceptions. I was able to > produce the same SSL read timeout when configuring an SSL Context Service > against a Redis instance that did not use TLS. However, when I removed the > SSl context service, it resumed working. > > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon > > > Virus-free. > www.avast.com > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link > > > <#m_3329527056917547118_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > On Fri, Aug 6, 2021 at 6:09 AM Jens M. Kofoed <jmkofoed....@gmail.com> > wrote: > > > I found the bug. - Reported as > > https://issues.apache.org/jira/browse/NIFI-9019 > > > > If I select database index 0, everything works fine. My process which is > > not working is using a database index of 3. > > So changing my small test flow (which is working) from database index 0 > -> > > 3 produce the same errors in the log and stop working. > > > > I don't know what should make this stop working, but I works fine in > > version 1.13.2 but not in version 1.14.0 > > Create a RedisConnectionPoolService with Database Index = 0 > > Test attach flow: everything should work > > Change Database Index = 3 > > tail -f the nifi-app.log file and start the the flow again. > > > > kind Regards > > Jens M. Kofoed > > > > Den fre. 6. aug. 2021 kl. 11.31 skrev Jens M. Kofoed < > > jmkofoed....@gmail.com>: > > > >> Some more digging > >> > >> I restarted my Redis cluster. Created a new RedisConnectionPoolService > >> and a new RedisDistributedMapCacheClientService. > >> I created a small test flow: Genereate flowfile -> > PutDistributedMapCache > >> -> AttributeUpdate > >> No problems, it works but... > >> > >> I went back to one the other flows where I use a DetectDuplicate process > >> and I got the following lines in the log > >> > >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] > >> o.a.n.r.s.RedisConnectionPoolService > >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] > >> Connecting to Redis in sentinel mode... > >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] > >> o.a.n.r.s.RedisConnectionPoolService > >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] > Redis > >> master = mymaster > >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] > >> o.a.n.r.s.RedisConnectionPoolService > >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] > Redis > >> sentinel at redis-01:26379 > >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] > >> o.a.n.r.s.RedisConnectionPoolService > >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] > Redis > >> sentinel at redis-02:26379 > >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] > >> o.a.n.r.s.RedisConnectionPoolService > >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] > Redis > >> sentinel at redis-03:26379 > >> 2021-08-06 11:18:21,124 INFO [Timer-Driven Process Thread-13] > >> redis.clients.jedis.JedisSentinelPool Trying to find master from > available > >> Sentinels... > >> 2021-08-06 11:18:21,124 DEBUG [Timer-Driven Process Thread-13] > >> redis.clients.jedis.JedisSentinelPool Connecting to Sentinel > redis-01:26379 > >> 2021-08-06 11:18:21,124 WARN [NiFi Web Server-4964] > >> org.apache.nifi.audit.ProcessorAuditor Unable to record actions: null > >> 2021-08-06 11:18:21,214 INFO [Flow Service Tasks Thread-1] > >> o.a.nifi.controller.StandardFlowService Saved flow controller > >> org.apache.nifi.controller.FlowController@33ab3784 // Another save > >> pending = false > >> 2021-08-06 11:18:31,134 WARN [Timer-Driven Process Thread-13] > >> redis.clients.jedis.JedisSentinelPool Cannot get master address from > >> sentinel running @ i-redis-01:26379. Reason: {}. Trying next one. > >> redis.clients.jedis.exceptions.JedisConnectionException: > >> javax.net.ssl.SSLException: Read timed out > >> at redis.clients.jedis.Connection.flush(Connection.java:342) > >> > >> So it connects to me Redis Sentinel cluster, but later it somehow trying > >> to use SSL or what??? > >> > >> kind regards > >> Jens M. Kofoed > >> > >> Den fre. 6. aug. 2021 kl. 08.07 skrev Jens M. Kofoed < > >> jmkofoed....@gmail.com>: > >> > >>> Hi > >>> > >>> I found another WARN in my log, which is about SSLException. However I > >>> did NOT specified the new property: SSL Context Service > >>> From the documentation: If specified, this service will be used to > >>> create an SSL Context that will be used to secure communications; if > not > >>> specified, communications will not be secure > >>> > >>> 2021-08-06 07:47:30,166 WARN [Timer-Driven Process Thread-12] > >>> redis.clients.jedis.JedisSentinelPool Cannot get master address from > >>> sentinel running @ redis-03:26379. Reason: {}. Trying next one. > >>> redis.clients.jedis.exceptions.JedisConnectionException: > >>> javax.net.ssl.SSLException: Read timed out > >>> at redis.clients.jedis.Connection.flush(Connection.java:342) > >>> at > >>> redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:269) > >>> at redis.clients.jedis.BinaryJedis.select(BinaryJedis.java:787) > >>> at > >>> > redis.clients.jedis.BinaryJedis.initializeFromClientConfig(BinaryJedis.java:99) > >>> at redis.clients.jedis.BinaryJedis.<init>(BinaryJedis.java:82) > >>> at redis.clients.jedis.Jedis.<init>(Jedis.java:54) > >>> at > >>> > redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:251) > >>> at > >>> > redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205) > >>> at > >>> > redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193) > >>> at > >>> > org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411) > >>> at > >>> > org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393) > >>> at > >>> > org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348) > >>> at > >>> > org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342) > >>> at > >>> > org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92) > >>> at sun.reflect.GeneratedMethodAccessor1212.invoke(Unknown > Source) > >>> at > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> at java.lang.reflect.Method.invoke(Method.java:498) > >>> at > >>> > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) > >>> at > >>> > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) > >>> at com.sun.proxy.$Proxy181.getConnection(Unknown Source) > >>> at > >>> > org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375) > >>> at > >>> > org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144) > >>> at sun.reflect.GeneratedMethodAccessor1211.invoke(Unknown > Source) > >>> at > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> at java.lang.reflect.Method.invoke(Method.java:498) > >>> at > >>> > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) > >>> at > >>> > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) > >>> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source) > >>> at > >>> > org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) > >>> at > >>> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > >>> at > >>> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) > >>> at > >>> > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) > >>> at > >>> > org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) > >>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > >>> at > >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >>> at > >>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > >>> at > >>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > >>> at > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >>> at > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >>> at java.lang.Thread.run(Thread.java:748) > >>> Caused by: javax.net.ssl.SSLException: Read timed out > >>> at sun.security.ssl.Alert.createSSLException(Alert.java:127) > >>> at > >>> sun.security.ssl.TransportContext.fatal(TransportContext.java:324) > >>> at > >>> sun.security.ssl.TransportContext.fatal(TransportContext.java:267) > >>> at > >>> sun.security.ssl.TransportContext.fatal(TransportContext.java:262) > >>> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:138) > >>> at > sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1383) > >>> at > >>> > sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1291) > >>> at > >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435) > >>> at > >>> sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:804) > >>> at > >>> sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:73) > >>> at > >>> > sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1166) > >>> at > >>> > redis.clients.jedis.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52) > >>> at > >>> > redis.clients.jedis.util.RedisOutputStream.flush(RedisOutputStream.java:133) > >>> at redis.clients.jedis.Connection.flush(Connection.java:339) > >>> ... 40 common frames omitted > >>> Caused by: java.net.SocketTimeoutException: Read timed out > >>> at java.net.SocketInputStream.socketRead0(Native Method) > >>> at > >>> java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > >>> at java.net.SocketInputStream.read(SocketInputStream.java:171) > >>> at java.net.SocketInputStream.read(SocketInputStream.java:141) > >>> at > >>> > sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457) > >>> at > >>> > sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165) > >>> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109) > >>> ... 49 common frames omitted > >>> > >>> Kind regards > >>> Jens M. Kofoed > >>> > >>> Den fre. 6. aug. 2021 kl. 06.35 skrev Jens M. Kofoed < > >>> jmkofoed....@gmail.com>: > >>> > >>>> Dear Develovper > >>>> > >>>> I'm using many RedisDistributedMapCacheClientService and > >>>> some RedisConnectionPoolService which connect to my 3 node cluster > >>>> with Redis sentinels. > >>>> After I upgraded one of my NiFi servers to 1.14.0 the processors which > >>>> is using the Redis start having issues. Looking in the logs I got the > >>>> following errors/WARN. > >>>> > >>>> 2021-08-05 13:24:12,556 WARN [Timer-Driven Process Thread-3] > >>>> o.a.n.controller.tasks.ConnectableTask Administratively Yielding > >>>> DetectDuplicate[id=feb1f2ff-4f9c-31d7-87c8-e225a80ef03f] due to > uncaught > >>>> Exception: redis.clients.jedis.exceptions.JedisConnectionException: > All > >>>> sentinels down, cannot determine where is mymaster master is > running... > >>>> redis.clients.jedis.exceptions.JedisConnectionException: All sentinels > >>>> down, cannot determine where is mymaster master is running... > >>>> at > >>>> > redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:281) > >>>> at > >>>> > redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205) > >>>> at > >>>> > redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193) > >>>> at > >>>> > org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411) > >>>> at > >>>> > org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393) > >>>> at > >>>> > org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348) > >>>> at > >>>> > org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342) > >>>> at > >>>> > org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92) > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>>> at > >>>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>>> at > >>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>> at java.lang.reflect.Method.invoke(Method.java:498) > >>>> at > >>>> > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) > >>>> at > >>>> > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) > >>>> at com.sun.proxy.$Proxy181.getConnection(Unknown Source) > >>>> at > >>>> > org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375) > >>>> at > >>>> > org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144) > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>>> at > >>>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>>> at > >>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>> at java.lang.reflect.Method.invoke(Method.java:498) > >>>> at > >>>> > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) > >>>> at > >>>> > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) > >>>> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source) > >>>> at > >>>> > org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) > >>>> at > >>>> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > >>>> at > >>>> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) > >>>> at > >>>> > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) > >>>> at > >>>> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) > >>>> at > org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > >>>> at > >>>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >>>> at > >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > >>>> at > >>>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > >>>> at > >>>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > >>>> at > >>>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >>>> at > >>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >>>> at java.lang.Thread.run(Thread.java:748) > >>>> > >>>> I know the WARN says: All sentinels down, cannot determine where is > >>>> mymaster master is running... BUT It is not. The Sentinel master is > >>>> running, I have a 2nd NIFI server running with v. 1.13.2 which is > using the > >>>> same Redis cluster and that one is still working. If I change the > server > >>>> (which has issues in v. 1.14.0) back to 1.13.2 there are no errors. I > also > >>>> tried to change one of the other servers to v. 1.14.0 and it have the > same > >>>> issue. > >>>> > >>>> A new feature in v.1.14.0 is the property "SSL Context > >>>> Service" RedisConnectionPoolService. It says if configured it will > use ssl > >>>> to establish a secure connection, if it's not configured it will not > be > >>>> secure. And It is not configured. > >>>> > >>>> Please any help :-) > >>>> > >>>> kind regards > >>>> Jens M. Kofoed > >>>> > >>>> > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon > > > Virus-free. > www.avast.com > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link > > > <#m_3329527056917547118_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >
