Thanks for the attached flow, Jens. So just to confirm, if you use Database Index 0, it works on both 1.13.2 and 1.14.0, but if you switch Database Index to 3, it only works on 1.13.2?
I was not able to reproduce this in Standalone mode, but I have not tried Sentinel mode. Can you verify whether Standalone mode works for you? It's still puzzling that there would be SSL exceptions. I was able to produce the same SSL read timeout when configuring an SSL Context Service against a Redis instance that did not use TLS. However, when I removed the SSl context service, it resumed working. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <#m_3329527056917547118_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> On Fri, Aug 6, 2021 at 6:09 AM Jens M. Kofoed <jmkofoed....@gmail.com> wrote: > I found the bug. - Reported as > https://issues.apache.org/jira/browse/NIFI-9019 > > If I select database index 0, everything works fine. My process which is > not working is using a database index of 3. > So changing my small test flow (which is working) from database index 0 -> > 3 produce the same errors in the log and stop working. > > I don't know what should make this stop working, but I works fine in > version 1.13.2 but not in version 1.14.0 > Create a RedisConnectionPoolService with Database Index = 0 > Test attach flow: everything should work > Change Database Index = 3 > tail -f the nifi-app.log file and start the the flow again. > > kind Regards > Jens M. Kofoed > > Den fre. 6. aug. 2021 kl. 11.31 skrev Jens M. Kofoed < > jmkofoed....@gmail.com>: > >> Some more digging >> >> I restarted my Redis cluster. Created a new RedisConnectionPoolService >> and a new RedisDistributedMapCacheClientService. >> I created a small test flow: Genereate flowfile -> PutDistributedMapCache >> -> AttributeUpdate >> No problems, it works but... >> >> I went back to one the other flows where I use a DetectDuplicate process >> and I got the following lines in the log >> >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >> o.a.n.r.s.RedisConnectionPoolService >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] >> Connecting to Redis in sentinel mode... >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >> o.a.n.r.s.RedisConnectionPoolService >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis >> master = mymaster >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >> o.a.n.r.s.RedisConnectionPoolService >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis >> sentinel at redis-01:26379 >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >> o.a.n.r.s.RedisConnectionPoolService >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis >> sentinel at redis-02:26379 >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >> o.a.n.r.s.RedisConnectionPoolService >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis >> sentinel at redis-03:26379 >> 2021-08-06 11:18:21,124 INFO [Timer-Driven Process Thread-13] >> redis.clients.jedis.JedisSentinelPool Trying to find master from available >> Sentinels... >> 2021-08-06 11:18:21,124 DEBUG [Timer-Driven Process Thread-13] >> redis.clients.jedis.JedisSentinelPool Connecting to Sentinel redis-01:26379 >> 2021-08-06 11:18:21,124 WARN [NiFi Web Server-4964] >> org.apache.nifi.audit.ProcessorAuditor Unable to record actions: null >> 2021-08-06 11:18:21,214 INFO [Flow Service Tasks Thread-1] >> o.a.nifi.controller.StandardFlowService Saved flow controller >> org.apache.nifi.controller.FlowController@33ab3784 // Another save >> pending = false >> 2021-08-06 11:18:31,134 WARN [Timer-Driven Process Thread-13] >> redis.clients.jedis.JedisSentinelPool Cannot get master address from >> sentinel running @ i-redis-01:26379. Reason: {}. Trying next one. >> redis.clients.jedis.exceptions.JedisConnectionException: >> javax.net.ssl.SSLException: Read timed out >> at redis.clients.jedis.Connection.flush(Connection.java:342) >> >> So it connects to me Redis Sentinel cluster, but later it somehow trying >> to use SSL or what??? >> >> kind regards >> Jens M. Kofoed >> >> Den fre. 6. aug. 2021 kl. 08.07 skrev Jens M. Kofoed < >> jmkofoed....@gmail.com>: >> >>> Hi >>> >>> I found another WARN in my log, which is about SSLException. However I >>> did NOT specified the new property: SSL Context Service >>> From the documentation: If specified, this service will be used to >>> create an SSL Context that will be used to secure communications; if not >>> specified, communications will not be secure >>> >>> 2021-08-06 07:47:30,166 WARN [Timer-Driven Process Thread-12] >>> redis.clients.jedis.JedisSentinelPool Cannot get master address from >>> sentinel running @ redis-03:26379. Reason: {}. Trying next one. >>> redis.clients.jedis.exceptions.JedisConnectionException: >>> javax.net.ssl.SSLException: Read timed out >>> at redis.clients.jedis.Connection.flush(Connection.java:342) >>> at >>> redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:269) >>> at redis.clients.jedis.BinaryJedis.select(BinaryJedis.java:787) >>> at >>> redis.clients.jedis.BinaryJedis.initializeFromClientConfig(BinaryJedis.java:99) >>> at redis.clients.jedis.BinaryJedis.<init>(BinaryJedis.java:82) >>> at redis.clients.jedis.Jedis.<init>(Jedis.java:54) >>> at >>> redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:251) >>> at >>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205) >>> at >>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193) >>> at >>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411) >>> at >>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393) >>> at >>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348) >>> at >>> org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342) >>> at >>> org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92) >>> at sun.reflect.GeneratedMethodAccessor1212.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >>> at >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >>> at com.sun.proxy.$Proxy181.getConnection(Unknown Source) >>> at >>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375) >>> at >>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144) >>> at sun.reflect.GeneratedMethodAccessor1211.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >>> at >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >>> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source) >>> at >>> org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) >>> at >>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >>> at >>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) >>> at >>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) >>> at >>> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) >>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: javax.net.ssl.SSLException: Read timed out >>> at sun.security.ssl.Alert.createSSLException(Alert.java:127) >>> at >>> sun.security.ssl.TransportContext.fatal(TransportContext.java:324) >>> at >>> sun.security.ssl.TransportContext.fatal(TransportContext.java:267) >>> at >>> sun.security.ssl.TransportContext.fatal(TransportContext.java:262) >>> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:138) >>> at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1383) >>> at >>> sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1291) >>> at >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435) >>> at >>> sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:804) >>> at >>> sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:73) >>> at >>> sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1166) >>> at >>> redis.clients.jedis.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52) >>> at >>> redis.clients.jedis.util.RedisOutputStream.flush(RedisOutputStream.java:133) >>> at redis.clients.jedis.Connection.flush(Connection.java:339) >>> ... 40 common frames omitted >>> Caused by: java.net.SocketTimeoutException: Read timed out >>> at java.net.SocketInputStream.socketRead0(Native Method) >>> at >>> java.net.SocketInputStream.socketRead(SocketInputStream.java:116) >>> at java.net.SocketInputStream.read(SocketInputStream.java:171) >>> at java.net.SocketInputStream.read(SocketInputStream.java:141) >>> at >>> sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457) >>> at >>> sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165) >>> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109) >>> ... 49 common frames omitted >>> >>> Kind regards >>> Jens M. Kofoed >>> >>> Den fre. 6. aug. 2021 kl. 06.35 skrev Jens M. Kofoed < >>> jmkofoed....@gmail.com>: >>> >>>> Dear Develovper >>>> >>>> I'm using many RedisDistributedMapCacheClientService and >>>> some RedisConnectionPoolService which connect to my 3 node cluster >>>> with Redis sentinels. >>>> After I upgraded one of my NiFi servers to 1.14.0 the processors which >>>> is using the Redis start having issues. Looking in the logs I got the >>>> following errors/WARN. >>>> >>>> 2021-08-05 13:24:12,556 WARN [Timer-Driven Process Thread-3] >>>> o.a.n.controller.tasks.ConnectableTask Administratively Yielding >>>> DetectDuplicate[id=feb1f2ff-4f9c-31d7-87c8-e225a80ef03f] due to uncaught >>>> Exception: redis.clients.jedis.exceptions.JedisConnectionException: All >>>> sentinels down, cannot determine where is mymaster master is running... >>>> redis.clients.jedis.exceptions.JedisConnectionException: All sentinels >>>> down, cannot determine where is mymaster master is running... >>>> at >>>> redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:281) >>>> at >>>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205) >>>> at >>>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193) >>>> at >>>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411) >>>> at >>>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393) >>>> at >>>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348) >>>> at >>>> org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342) >>>> at >>>> org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>> at >>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >>>> at >>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >>>> at com.sun.proxy.$Proxy181.getConnection(Unknown Source) >>>> at >>>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375) >>>> at >>>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>> at >>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >>>> at >>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >>>> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source) >>>> at >>>> org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) >>>> at >>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >>>> at >>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) >>>> at >>>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) >>>> at >>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) >>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>> at >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>> at java.lang.Thread.run(Thread.java:748) >>>> >>>> I know the WARN says: All sentinels down, cannot determine where is >>>> mymaster master is running... BUT It is not. The Sentinel master is >>>> running, I have a 2nd NIFI server running with v. 1.13.2 which is using the >>>> same Redis cluster and that one is still working. If I change the server >>>> (which has issues in v. 1.14.0) back to 1.13.2 there are no errors. I also >>>> tried to change one of the other servers to v. 1.14.0 and it have the same >>>> issue. >>>> >>>> A new feature in v.1.14.0 is the property "SSL Context >>>> Service" RedisConnectionPoolService. It says if configured it will use ssl >>>> to establish a secure connection, if it's not configured it will not be >>>> secure. And It is not configured. >>>> >>>> Please any help :-) >>>> >>>> kind regards >>>> Jens M. Kofoed >>>> >>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <#m_3329527056917547118_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
