Correcting last line: This is a 3 node unsecure Redis/Sentinel (without SSL).
On Fri, Aug 6, 2021 at 8:46 AM Eduardo Fontes <eduardo.fon...@gmail.com> wrote: > Hi Joe, > > I can reproduce the error. > I have a three node secured NiFi 1.14. I've tested with DetectDuplicate > and it only worked with db index 0 in RedisConnectionPoolService. If any > other index, failed with message: > > DetectDuplicate[id=c4a64927-6056-1d2d-ffff-ffffa3ac489e] Failed to process > session due to All sentinels down, cannot determine where is clnifi master > is running...; Processor Administratively Yielded for 1 sec: > redis.clients.jedis.exceptions.JedisConnectionException: All sentinels > down, cannot determine where is clnifi master is running... > > This is a 3 node Redis/Sentinel without unsecure (without SSL). > > Eduardo Fontes > > On Fri, Aug 6, 2021 at 8:09 AM Joe Gresock <jgres...@gmail.com> wrote: > >> Thanks for the attached flow, Jens. So just to confirm, if you use >> Database Index 0, it works on both 1.13.2 and 1.14.0, but if you switch >> Database Index to 3, it only works on 1.13.2? >> >> I was not able to reproduce this in Standalone mode, but I have not tried >> Sentinel mode. Can you verify whether Standalone mode works for you? >> >> It's still puzzling that there would be SSL exceptions. I was able to >> produce the same SSL read timeout when configuring an SSL Context Service >> against a Redis instance that did not use TLS. However, when I removed >> the >> SSl context service, it resumed working. >> >> < >> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon >> > >> Virus-free. >> www.avast.com >> < >> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link >> > >> <#m_3329527056917547118_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >> >> On Fri, Aug 6, 2021 at 6:09 AM Jens M. Kofoed <jmkofoed....@gmail.com> >> wrote: >> >> > I found the bug. - Reported as >> > https://issues.apache.org/jira/browse/NIFI-9019 >> > >> > If I select database index 0, everything works fine. My process which is >> > not working is using a database index of 3. >> > So changing my small test flow (which is working) from database index 0 >> -> >> > 3 produce the same errors in the log and stop working. >> > >> > I don't know what should make this stop working, but I works fine in >> > version 1.13.2 but not in version 1.14.0 >> > Create a RedisConnectionPoolService with Database Index = 0 >> > Test attach flow: everything should work >> > Change Database Index = 3 >> > tail -f the nifi-app.log file and start the the flow again. >> > >> > kind Regards >> > Jens M. Kofoed >> > >> > Den fre. 6. aug. 2021 kl. 11.31 skrev Jens M. Kofoed < >> > jmkofoed....@gmail.com>: >> > >> >> Some more digging >> >> >> >> I restarted my Redis cluster. Created a new RedisConnectionPoolService >> >> and a new RedisDistributedMapCacheClientService. >> >> I created a small test flow: Genereate flowfile -> >> PutDistributedMapCache >> >> -> AttributeUpdate >> >> No problems, it works but... >> >> >> >> I went back to one the other flows where I use a DetectDuplicate >> process >> >> and I got the following lines in the log >> >> >> >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >> >> o.a.n.r.s.RedisConnectionPoolService >> >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] >> >> Connecting to Redis in sentinel mode... >> >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >> >> o.a.n.r.s.RedisConnectionPoolService >> >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] >> Redis >> >> master = mymaster >> >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >> >> o.a.n.r.s.RedisConnectionPoolService >> >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] >> Redis >> >> sentinel at redis-01:26379 >> >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >> >> o.a.n.r.s.RedisConnectionPoolService >> >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] >> Redis >> >> sentinel at redis-02:26379 >> >> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13] >> >> o.a.n.r.s.RedisConnectionPoolService >> >> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] >> Redis >> >> sentinel at redis-03:26379 >> >> 2021-08-06 11:18:21,124 INFO [Timer-Driven Process Thread-13] >> >> redis.clients.jedis.JedisSentinelPool Trying to find master from >> available >> >> Sentinels... >> >> 2021-08-06 11:18:21,124 DEBUG [Timer-Driven Process Thread-13] >> >> redis.clients.jedis.JedisSentinelPool Connecting to Sentinel >> redis-01:26379 >> >> 2021-08-06 11:18:21,124 WARN [NiFi Web Server-4964] >> >> org.apache.nifi.audit.ProcessorAuditor Unable to record actions: null >> >> 2021-08-06 11:18:21,214 INFO [Flow Service Tasks Thread-1] >> >> o.a.nifi.controller.StandardFlowService Saved flow controller >> >> org.apache.nifi.controller.FlowController@33ab3784 // Another save >> >> pending = false >> >> 2021-08-06 11:18:31,134 WARN [Timer-Driven Process Thread-13] >> >> redis.clients.jedis.JedisSentinelPool Cannot get master address from >> >> sentinel running @ i-redis-01:26379. Reason: {}. Trying next one. >> >> redis.clients.jedis.exceptions.JedisConnectionException: >> >> javax.net.ssl.SSLException: Read timed out >> >> at redis.clients.jedis.Connection.flush(Connection.java:342) >> >> >> >> So it connects to me Redis Sentinel cluster, but later it somehow >> trying >> >> to use SSL or what??? >> >> >> >> kind regards >> >> Jens M. Kofoed >> >> >> >> Den fre. 6. aug. 2021 kl. 08.07 skrev Jens M. Kofoed < >> >> jmkofoed....@gmail.com>: >> >> >> >>> Hi >> >>> >> >>> I found another WARN in my log, which is about SSLException. However I >> >>> did NOT specified the new property: SSL Context Service >> >>> From the documentation: If specified, this service will be used to >> >>> create an SSL Context that will be used to secure communications; if >> not >> >>> specified, communications will not be secure >> >>> >> >>> 2021-08-06 07:47:30,166 WARN [Timer-Driven Process Thread-12] >> >>> redis.clients.jedis.JedisSentinelPool Cannot get master address from >> >>> sentinel running @ redis-03:26379. Reason: {}. Trying next one. >> >>> redis.clients.jedis.exceptions.JedisConnectionException: >> >>> javax.net.ssl.SSLException: Read timed out >> >>> at redis.clients.jedis.Connection.flush(Connection.java:342) >> >>> at >> >>> redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:269) >> >>> at >> redis.clients.jedis.BinaryJedis.select(BinaryJedis.java:787) >> >>> at >> >>> >> redis.clients.jedis.BinaryJedis.initializeFromClientConfig(BinaryJedis.java:99) >> >>> at redis.clients.jedis.BinaryJedis.<init>(BinaryJedis.java:82) >> >>> at redis.clients.jedis.Jedis.<init>(Jedis.java:54) >> >>> at >> >>> >> redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:251) >> >>> at >> >>> >> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205) >> >>> at >> >>> >> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193) >> >>> at >> >>> >> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411) >> >>> at >> >>> >> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393) >> >>> at >> >>> >> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348) >> >>> at >> >>> >> org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342) >> >>> at >> >>> >> org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92) >> >>> at sun.reflect.GeneratedMethodAccessor1212.invoke(Unknown >> Source) >> >>> at >> >>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>> at java.lang.reflect.Method.invoke(Method.java:498) >> >>> at >> >>> >> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >> >>> at >> >>> >> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >> >>> at com.sun.proxy.$Proxy181.getConnection(Unknown Source) >> >>> at >> >>> >> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375) >> >>> at >> >>> >> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144) >> >>> at sun.reflect.GeneratedMethodAccessor1211.invoke(Unknown >> Source) >> >>> at >> >>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>> at java.lang.reflect.Method.invoke(Method.java:498) >> >>> at >> >>> >> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >> >>> at >> >>> >> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >> >>> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source) >> >>> at >> >>> >> org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) >> >>> at >> >>> >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> >>> at >> >>> >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) >> >>> at >> >>> >> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) >> >>> at >> >>> >> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) >> >>> at >> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >> >>> at >> >>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> at >> >>> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >> >>> at >> >>> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >>> at java.lang.Thread.run(Thread.java:748) >> >>> Caused by: javax.net.ssl.SSLException: Read timed out >> >>> at sun.security.ssl.Alert.createSSLException(Alert.java:127) >> >>> at >> >>> sun.security.ssl.TransportContext.fatal(TransportContext.java:324) >> >>> at >> >>> sun.security.ssl.TransportContext.fatal(TransportContext.java:267) >> >>> at >> >>> sun.security.ssl.TransportContext.fatal(TransportContext.java:262) >> >>> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:138) >> >>> at >> sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1383) >> >>> at >> >>> >> sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1291) >> >>> at >> >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435) >> >>> at >> >>> >> sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:804) >> >>> at >> >>> sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:73) >> >>> at >> >>> >> sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1166) >> >>> at >> >>> >> redis.clients.jedis.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52) >> >>> at >> >>> >> redis.clients.jedis.util.RedisOutputStream.flush(RedisOutputStream.java:133) >> >>> at redis.clients.jedis.Connection.flush(Connection.java:339) >> >>> ... 40 common frames omitted >> >>> Caused by: java.net.SocketTimeoutException: Read timed out >> >>> at java.net.SocketInputStream.socketRead0(Native Method) >> >>> at >> >>> java.net.SocketInputStream.socketRead(SocketInputStream.java:116) >> >>> at java.net.SocketInputStream.read(SocketInputStream.java:171) >> >>> at java.net.SocketInputStream.read(SocketInputStream.java:141) >> >>> at >> >>> >> sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457) >> >>> at >> >>> >> sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165) >> >>> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109) >> >>> ... 49 common frames omitted >> >>> >> >>> Kind regards >> >>> Jens M. Kofoed >> >>> >> >>> Den fre. 6. aug. 2021 kl. 06.35 skrev Jens M. Kofoed < >> >>> jmkofoed....@gmail.com>: >> >>> >> >>>> Dear Develovper >> >>>> >> >>>> I'm using many RedisDistributedMapCacheClientService and >> >>>> some RedisConnectionPoolService which connect to my 3 node cluster >> >>>> with Redis sentinels. >> >>>> After I upgraded one of my NiFi servers to 1.14.0 the processors >> which >> >>>> is using the Redis start having issues. Looking in the logs I got the >> >>>> following errors/WARN. >> >>>> >> >>>> 2021-08-05 13:24:12,556 WARN [Timer-Driven Process Thread-3] >> >>>> o.a.n.controller.tasks.ConnectableTask Administratively Yielding >> >>>> DetectDuplicate[id=feb1f2ff-4f9c-31d7-87c8-e225a80ef03f] due to >> uncaught >> >>>> Exception: redis.clients.jedis.exceptions.JedisConnectionException: >> All >> >>>> sentinels down, cannot determine where is mymaster master is >> running... >> >>>> redis.clients.jedis.exceptions.JedisConnectionException: All >> sentinels >> >>>> down, cannot determine where is mymaster master is running... >> >>>> at >> >>>> >> redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:281) >> >>>> at >> >>>> >> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205) >> >>>> at >> >>>> >> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193) >> >>>> at >> >>>> >> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411) >> >>>> at >> >>>> >> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393) >> >>>> at >> >>>> >> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348) >> >>>> at >> >>>> >> org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342) >> >>>> at >> >>>> >> org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92) >> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> >>>> at >> >>>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >>>> at >> >>>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>>> at java.lang.reflect.Method.invoke(Method.java:498) >> >>>> at >> >>>> >> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >> >>>> at >> >>>> >> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >> >>>> at com.sun.proxy.$Proxy181.getConnection(Unknown Source) >> >>>> at >> >>>> >> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375) >> >>>> at >> >>>> >> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144) >> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> >>>> at >> >>>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >>>> at >> >>>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>>> at java.lang.reflect.Method.invoke(Method.java:498) >> >>>> at >> >>>> >> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) >> >>>> at >> >>>> >> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105) >> >>>> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source) >> >>>> at >> >>>> >> org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) >> >>>> at >> >>>> >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> >>>> at >> >>>> >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) >> >>>> at >> >>>> >> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) >> >>>> at >> >>>> >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) >> >>>> at >> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >> >>>> at >> >>>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >>>> at >> >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> >>>> at >> >>>> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >> >>>> at >> >>>> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >> >>>> at >> >>>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >>>> at >> >>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >>>> at java.lang.Thread.run(Thread.java:748) >> >>>> >> >>>> I know the WARN says: All sentinels down, cannot determine where is >> >>>> mymaster master is running... BUT It is not. The Sentinel master is >> >>>> running, I have a 2nd NIFI server running with v. 1.13.2 which is >> using the >> >>>> same Redis cluster and that one is still working. If I change the >> server >> >>>> (which has issues in v. 1.14.0) back to 1.13.2 there are no errors. >> I also >> >>>> tried to change one of the other servers to v. 1.14.0 and it have >> the same >> >>>> issue. >> >>>> >> >>>> A new feature in v.1.14.0 is the property "SSL Context >> >>>> Service" RedisConnectionPoolService. It says if configured it will >> use ssl >> >>>> to establish a secure connection, if it's not configured it will not >> be >> >>>> secure. And It is not configured. >> >>>> >> >>>> Please any help :-) >> >>>> >> >>>> kind regards >> >>>> Jens M. Kofoed >> >>>> >> >>>> >> < >> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon >> > >> Virus-free. >> www.avast.com >> < >> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link >> > >> <#m_3329527056917547118_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >> >
