Yes. With database index 3 it works in v. 1.13.2 but NOT in v. 1.14.0 I don’t have the possibility to test the standalone. But Eduardo just confirmed it doesn’t work for sentinels in v. 1.14.0
Kind regards Jens M. Kofoed > Den 6. aug. 2021 kl. 13.49 skrev Eduardo Fontes <eduardo.fon...@gmail.com>: > > Correcting last line: > > This is a 3 node unsecure Redis/Sentinel (without SSL). > > On Fri, Aug 6, 2021 at 8:46 AM Eduardo Fontes <eduardo.fon...@gmail.com> > wrote: > >> Hi Joe, >> >> I can reproduce the error. >> I have a three node secured NiFi 1.14. I've tested with DetectDuplicate >> and it only worked with db index 0 in RedisConnectionPoolService. If any >> other index, failed with message: >> >> DetectDuplicate[id=c4a64927-6056-1d2d-ffff-ffffa3ac489e] Failed to process >> session due to All sentinels down, cannot determine where is clnifi master >> is running...; Processor Administratively Yielded for 1 sec: >> redis.clients.jedis.exceptions.JedisConnectionException: All sentinels >> down, cannot determine where is clnifi master is running... >> >> This is a 3 node Redis/Sentinel without unsecure (without SSL). >> >> Eduardo Fontes >> >>> On Fri, Aug 6, 2021 at 8:09 AM Joe Gresock <jgres...@gmail.com> wrote: >>> >>> Thanks for the attached flow, Jens. So just to confirm, if you use >>> Database Index 0, it works on both 1.13.2 and 1.14.0, but if you switch >>> Database Index to 3, it only works on 1.13.2? >>> >>> I was not able to reproduce this in Standalone mode, but I have not tried >>> Sentinel mode. Can you verify whether Standalone mode works for you? >>> >>> It's still puzzling that there would be SSL exceptions. I was able to >>> produce the same SSL read timeout when configuring an SSL Context Service >>> against a Redis instance that did not use TLS. However, when I removed >>> the >>> SSl context service, it resumed working. >>> >>> < >>> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon >>>> >>> Virus-free. >>> www.avast.com >>> < >>> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link >>>> >>> <#m_3329527056917547118_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>> >>> On Fri, Aug 6, 2021 at 6:09 AM Jens M. Kofoed <jmkofoed....@gmail.com> >>> wrote: >>> >>>> I found the bug. - Reported as >>>> https://issues.apache.org/jira/browse/NIFI-9019 >>>> >>>> If I select database index 0, everything works fine. My process which is >>>> not working is using a database index of 3. >>>> So changing my small test flow (which is working) from database index 0 >>> -> >>>> 3 produce the same errors in the log and stop working. >>>> >>>> I don't know what should make this stop working, but I works fine in >>>> version 1.13.2 but not in version 1.14.0 >>>> Create a RedisConnectionPoolService with Database Index = 0 >>>> Test attach flow: everything should work >>>> Change Database Index = 3 >>>> tail -f the nifi-app.log file and start the the flow again. >>>> >>>> kind Regards >>>> Jens M. Kofoed >>>> >>>> Den fre. 6. aug. 2021 kl. 11.31 skrev Jens M. Kofoed < >>>> jmkofoed....@gmail.com>: >>>> >>>>> Some more digging >>>>> >>>>> I restarted my Redis cluster. Created a new RedisConnectionPoolService >>>>> and a new RedisDistributedMapCacheClientService. >>>>> I created a small test flow: Genereate flowfile -> >>> PutDistributedMapCache >>>>> -> AttributeUpdate >>>>> No problems, it works but... >>>>> >>>>> I went back to one the other flows where I use a DetectDuplicate >>> process >>>>> and I got the following lines in the log >>>>> >>>>> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >>>>> o.a.n.r.s.RedisConnectionPoolService >>>>> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] >>>>> Connecting to Redis in sentinel mode... >>>>> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >>>>> o.a.n.r.s.RedisConnectionPoolService >>>>> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] >>> Redis >>>>> master = mymaster >>>>> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >>>>> o.a.n.r.s.RedisConnectionPoolService >>>>> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] >>> Redis >>>>> sentinel at redis-01:26379 >>>>> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >>>>> o.a.n.r.s.RedisConnectionPoolService >>>>> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] >>> Redis >>>>> sentinel at redis-02:26379 >>>>> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >>>>> o.a.n.r.s.RedisConnectionPoolService >>>>> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] >>> Redis >>>>> sentinel at redis-03:26379 >>>>> 2021-08-06 11:18:21,124 INFO [Timer-Driven Process Thread-13] >>>>> redis.clients.jedis.JedisSentinelPool Trying to find master from >>> available >>>>> Sentinels... >>>>> 2021-08-06 11:18:21,124 DEBUG [Timer-Driven Process Thread-13] >>>>> redis.clients.jedis.JedisSentinelPool Connecting to Sentinel >>> redis-01:26379 >>>>> 2021-08-06 11:18:21,124 WARN [NiFi Web Server-4964] >>>>> org.apache.nifi.audit.ProcessorAuditor Unable to record actions: null >>>>> 2021-08-06 11:18:21,214 INFO [Flow Service Tasks Thread-1] >>>>> o.a.nifi.controller.StandardFlowService Saved flow controller >>>>> org.apache.nifi.controller.FlowController@33ab3784 // Another save >>>>> pending = false >>>>> 2021-08-06 11:18:31,134 WARN [Timer-Driven Process Thread-13] >>>>> redis.clients.jedis.JedisSentinelPool Cannot get master address from >>>>> sentinel running @ i-redis-01:26379. Reason: {}. Trying next one. >>>>> redis.clients.jedis.exceptions.JedisConnectionException: >>>>> javax.net.ssl.SSLException: Read timed out >>>>> at redis.clients.jedis.Connection.flush(Connection.java:342) >>>>> >>>>> So it connects to me Redis Sentinel cluster, but later it somehow >>> trying >>>>> to use SSL or what??? >>>>> >>>>> kind regards >>>>> Jens M. Kofoed >>>>> >>>>> Den fre. 6. aug. 2021 kl. 08.07 skrev Jens M. Kofoed < >>>>> jmkofoed....@gmail.com>: >>>>> >>>>>> Hi >>>>>> >>>>>> I found another WARN in my log, which is about SSLException. However I >>>>>> did NOT specified the new property: SSL Context Service >>>>>> From the documentation: If specified, this service will be used to >>>>>> create an SSL Context that will be used to secure communications; if >>> not >>>>>> specified, communications will not be secure >>>>>> >>>>>> 2021-08-06 07:47:30,166 WARN [Timer-Driven Process Thread-12] >>>>>> redis.clients.jedis.JedisSentinelPool Cannot get master address from >>>>>> sentinel running @ redis-03:26379. Reason: {}. Trying next one. >>>>>> redis.clients.jedis.exceptions.JedisConnectionException: >>>>>> javax.net.ssl.SSLException: Read timed out >>>>>> at redis.clients.jedis.Connection.flush(Connection.java:342) >>>>>> at >>>>>> redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:269) >>>>>> at >>> redis.clients.jedis.BinaryJedis.select(BinaryJedis.java:787) >>>>>> at >>>>>> >>> redis.clients.jedis.BinaryJedis.initializeFromClientConfig(BinaryJedis.java:99) >>>>>> at redis.clients.jedis.BinaryJedis.<init>(BinaryJedis.java:82) >>>>>> at redis.clients.jedis.Jedis.<init>(Jedis.java:54) >>>>>> at >>>>>> >>> redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:251) >>>>>> at >>>>>> >>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205) >>>>>> at >>>>>> >>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193) >>>>>> at >>>>>> >>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411) >>>>>> at >>>>>> >>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393) >>>>>> at >>>>>> >>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348) >>>>>> at >>>>>> >>> org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342) >>>>>> at >>>>>> >>> org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92) >>>>>> at sun.reflect.GeneratedMethodAccessor1212.invoke(Unknown >>> Source) >>>>>> at >>>>>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>> at >>>>>> >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >>>>>> at >>>>>> >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >>>>>> at com.sun.proxy.$Proxy181.getConnection(Unknown Source) >>>>>> at >>>>>> >>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375) >>>>>> at >>>>>> >>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144) >>>>>> at sun.reflect.GeneratedMethodAccessor1211.invoke(Unknown >>> Source) >>>>>> at >>>>>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>> at >>>>>> >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >>>>>> at >>>>>> >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >>>>>> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source) >>>>>> at >>>>>> >>> org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) >>>>>> at >>>>>> >>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >>>>>> at >>>>>> >>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) >>>>>> at >>>>>> >>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) >>>>>> at >>>>>> >>> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) >>>>>> at >>> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >>>>>> at >>>>>> >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>> at >>>>>> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >>>>>> at >>>>>> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>>>> at >>>>>> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>> at >>>>>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>> Caused by: javax.net.ssl.SSLException: Read timed out >>>>>> at sun.security.ssl.Alert.createSSLException(Alert.java:127) >>>>>> at >>>>>> sun.security.ssl.TransportContext.fatal(TransportContext.java:324) >>>>>> at >>>>>> sun.security.ssl.TransportContext.fatal(TransportContext.java:267) >>>>>> at >>>>>> sun.security.ssl.TransportContext.fatal(TransportContext.java:262) >>>>>> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:138) >>>>>> at >>> sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1383) >>>>>> at >>>>>> >>> sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1291) >>>>>> at >>>>>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435) >>>>>> at >>>>>> >>> sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:804) >>>>>> at >>>>>> sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:73) >>>>>> at >>>>>> >>> sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1166) >>>>>> at >>>>>> >>> redis.clients.jedis.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52) >>>>>> at >>>>>> >>> redis.clients.jedis.util.RedisOutputStream.flush(RedisOutputStream.java:133) >>>>>> at redis.clients.jedis.Connection.flush(Connection.java:339) >>>>>> ... 40 common frames omitted >>>>>> Caused by: java.net.SocketTimeoutException: Read timed out >>>>>> at java.net.SocketInputStream.socketRead0(Native Method) >>>>>> at >>>>>> java.net.SocketInputStream.socketRead(SocketInputStream.java:116) >>>>>> at java.net.SocketInputStream.read(SocketInputStream.java:171) >>>>>> at java.net.SocketInputStream.read(SocketInputStream.java:141) >>>>>> at >>>>>> >>> sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457) >>>>>> at >>>>>> >>> sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165) >>>>>> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109) >>>>>> ... 49 common frames omitted >>>>>> >>>>>> Kind regards >>>>>> Jens M. Kofoed >>>>>> >>>>>> Den fre. 6. aug. 2021 kl. 06.35 skrev Jens M. Kofoed < >>>>>> jmkofoed....@gmail.com>: >>>>>> >>>>>>> Dear Develovper >>>>>>> >>>>>>> I'm using many RedisDistributedMapCacheClientService and >>>>>>> some RedisConnectionPoolService which connect to my 3 node cluster >>>>>>> with Redis sentinels. >>>>>>> After I upgraded one of my NiFi servers to 1.14.0 the processors >>> which >>>>>>> is using the Redis start having issues. Looking in the logs I got the >>>>>>> following errors/WARN. >>>>>>> >>>>>>> 2021-08-05 13:24:12,556 WARN [Timer-Driven Process Thread-3] >>>>>>> o.a.n.controller.tasks.ConnectableTask Administratively Yielding >>>>>>> DetectDuplicate[id=feb1f2ff-4f9c-31d7-87c8-e225a80ef03f] due to >>> uncaught >>>>>>> Exception: redis.clients.jedis.exceptions.JedisConnectionException: >>> All >>>>>>> sentinels down, cannot determine where is mymaster master is >>> running... >>>>>>> redis.clients.jedis.exceptions.JedisConnectionException: All >>> sentinels >>>>>>> down, cannot determine where is mymaster master is running... >>>>>>> at >>>>>>> >>> redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:281) >>>>>>> at >>>>>>> >>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205) >>>>>>> at >>>>>>> >>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193) >>>>>>> at >>>>>>> >>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411) >>>>>>> at >>>>>>> >>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393) >>>>>>> at >>>>>>> >>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348) >>>>>>> at >>>>>>> >>> org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342) >>>>>>> at >>>>>>> >>> org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92) >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>>>>>> at >>>>>>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>> at >>>>>>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>>> at >>>>>>> >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >>>>>>> at >>>>>>> >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >>>>>>> at com.sun.proxy.$Proxy181.getConnection(Unknown Source) >>>>>>> at >>>>>>> >>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375) >>>>>>> at >>>>>>> >>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144) >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>>>>>> at >>>>>>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>> at >>>>>>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>>> at >>>>>>> >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >>>>>>> at >>>>>>> >>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >>>>>>> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source) >>>>>>> at >>>>>>> >>> org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) >>>>>>> at >>>>>>> >>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >>>>>>> at >>>>>>> >>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) >>>>>>> at >>>>>>> >>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) >>>>>>> at >>>>>>> >>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) >>>>>>> at >>> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >>>>>>> at >>>>>>> >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>> at >>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >>>>>>> at >>>>>>> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >>>>>>> at >>>>>>> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >>>>>>> at >>>>>>> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>>> at >>>>>>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>>> >>>>>>> I know the WARN says: All sentinels down, cannot determine where is >>>>>>> mymaster master is running... BUT It is not. The Sentinel master is >>>>>>> running, I have a 2nd NIFI server running with v. 1.13.2 which is >>> using the >>>>>>> same Redis cluster and that one is still working. If I change the >>> server >>>>>>> (which has issues in v. 1.14.0) back to 1.13.2 there are no errors. >>> I also >>>>>>> tried to change one of the other servers to v. 1.14.0 and it have >>> the same >>>>>>> issue. >>>>>>> >>>>>>> A new feature in v.1.14.0 is the property "SSL Context >>>>>>> Service" RedisConnectionPoolService. It says if configured it will >>> use ssl >>>>>>> to establish a secure connection, if it's not configured it will not >>> be >>>>>>> secure. And It is not configured. >>>>>>> >>>>>>> Please any help :-) >>>>>>> >>>>>>> kind regards >>>>>>> Jens M. Kofoed >>>>>>> >>>>>>> >>> < >>> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon >>>> >>> Virus-free. >>> www.avast.com >>> < >>> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link >>>> >>> <#m_3329527056917547118_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>> >>
