Some more digging
I restarted my Redis cluster. Created a new RedisConnectionPoolService and
a new RedisDistributedMapCacheClientService.
I created a small test flow: Genereate flowfile -> PutDistributedMapCache
-> AttributeUpdate
No problems, it works but...
I went back to one the other flows where I use a DetectDuplicate process
and I got the following lines in the log
2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13]
o.a.n.r.s.RedisConnectionPoolService
RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b]
Connecting to Redis in sentinel mode...
2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13]
o.a.n.r.s.RedisConnectionPoolService
RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis
master = mymaster
2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13]
o.a.n.r.s.RedisConnectionPoolService
RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis
sentinel at redis-01:26379
2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13]
o.a.n.r.s.RedisConnectionPoolService
RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis
sentinel at redis-02:26379
2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13]
o.a.n.r.s.RedisConnectionPoolService
RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis
sentinel at redis-03:26379
2021-08-06 11:18:21,124 INFO [Timer-Driven Process Thread-13]
redis.clients.jedis.JedisSentinelPool Trying to find master from available
Sentinels...
2021-08-06 11:18:21,124 DEBUG [Timer-Driven Process Thread-13]
redis.clients.jedis.JedisSentinelPool Connecting to Sentinel redis-01:26379
2021-08-06 11:18:21,124 WARN [NiFi Web Server-4964]
org.apache.nifi.audit.ProcessorAuditor Unable to record actions: null
2021-08-06 11:18:21,214 INFO [Flow Service Tasks Thread-1]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@33ab3784 // Another save pending
= false
2021-08-06 11:18:31,134 WARN [Timer-Driven Process Thread-13]
redis.clients.jedis.JedisSentinelPool Cannot get master address from
sentinel running @ i-redis-01:26379. Reason: {}. Trying next one.
redis.clients.jedis.exceptions.JedisConnectionException:
javax.net.ssl.SSLException: Read timed out
at redis.clients.jedis.Connection.flush(Connection.java:342)
So it connects to me Redis Sentinel cluster, but later it somehow trying to
use SSL or what???
kind regards
Jens M. Kofoed
Den fre. 6. aug. 2021 kl. 08.07 skrev Jens M. Kofoed <jmkofoed....@gmail.com
>:
> Hi
>
> I found another WARN in my log, which is about SSLException. However I did
> NOT specified the new property: SSL Context Service
> From the documentation: If specified, this service will be used to create
> an SSL Context that will be used to secure communications; if not
> specified, communications will not be secure
>
> 2021-08-06 07:47:30,166 WARN [Timer-Driven Process Thread-12]
> redis.clients.jedis.JedisSentinelPool Cannot get master address from
> sentinel running @ redis-03:26379. Reason: {}. Trying next one.
> redis.clients.jedis.exceptions.JedisConnectionException:
> javax.net.ssl.SSLException: Read timed out
> at redis.clients.jedis.Connection.flush(Connection.java:342)
> at
> redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:269)
> at redis.clients.jedis.BinaryJedis.select(BinaryJedis.java:787)
> at
> redis.clients.jedis.BinaryJedis.initializeFromClientConfig(BinaryJedis.java:99)
> at redis.clients.jedis.BinaryJedis.<init>(BinaryJedis.java:82)
> at redis.clients.jedis.Jedis.<init>(Jedis.java:54)
> at
> redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:251)
> at
> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205)
> at
> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193)
> at
> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411)
> at
> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393)
> at
> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348)
> at
> org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342)
> at
> org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92)
> at sun.reflect.GeneratedMethodAccessor1212.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
> at com.sun.proxy.$Proxy181.getConnection(Unknown Source)
> at
> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375)
> at
> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144)
> at sun.reflect.GeneratedMethodAccessor1211.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source)
> at
> org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
> at
> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.net.ssl.SSLException: Read timed out
> at sun.security.ssl.Alert.createSSLException(Alert.java:127)
> at
> sun.security.ssl.TransportContext.fatal(TransportContext.java:324)
> at
> sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
> at
> sun.security.ssl.TransportContext.fatal(TransportContext.java:262)
> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:138)
> at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1383)
> at
> sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1291)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435)
> at
> sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:804)
> at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:73)
> at
> sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1166)
> at
> redis.clients.jedis.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52)
> at
> redis.clients.jedis.util.RedisOutputStream.flush(RedisOutputStream.java:133)
> at redis.clients.jedis.Connection.flush(Connection.java:339)
> ... 40 common frames omitted
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at
> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at
> sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
> at
> sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165)
> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109)
> ... 49 common frames omitted
>
> Kind regards
> Jens M. Kofoed
>
> Den fre. 6. aug. 2021 kl. 06.35 skrev Jens M. Kofoed <
> jmkofoed....@gmail.com>:
>
>> Dear Develovper
>>
>> I'm using many RedisDistributedMapCacheClientService and
>> some RedisConnectionPoolService which connect to my 3 node cluster
>> with Redis sentinels.
>> After I upgraded one of my NiFi servers to 1.14.0 the processors which is
>> using the Redis start having issues. Looking in the logs I got the
>> following errors/WARN.
>>
>> 2021-08-05 13:24:12,556 WARN [Timer-Driven Process Thread-3]
>> o.a.n.controller.tasks.ConnectableTask Administratively Yielding
>> DetectDuplicate[id=feb1f2ff-4f9c-31d7-87c8-e225a80ef03f] due to uncaught
>> Exception: redis.clients.jedis.exceptions.JedisConnectionException: All
>> sentinels down, cannot determine where is mymaster master is running...
>> redis.clients.jedis.exceptions.JedisConnectionException: All sentinels
>> down, cannot determine where is mymaster master is running...
>> at
>> redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:281)
>> at
>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205)
>> at
>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193)
>> at
>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411)
>> at
>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393)
>> at
>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348)
>> at
>> org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342)
>> at
>> org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
>> at
>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
>> at com.sun.proxy.$Proxy181.getConnection(Unknown Source)
>> at
>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375)
>> at
>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
>> at
>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
>> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source)
>> at
>> org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183)
>> at
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
>> at
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
>> at
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> I know the WARN says: All sentinels down, cannot determine where is
>> mymaster master is running... BUT It is not. The Sentinel master is
>> running, I have a 2nd NIFI server running with v. 1.13.2 which is using the
>> same Redis cluster and that one is still working. If I change the server
>> (which has issues in v. 1.14.0) back to 1.13.2 there are no errors. I also
>> tried to change one of the other servers to v. 1.14.0 and it have the same
>> issue.
>>
>> A new feature in v.1.14.0 is the property "SSL Context
>> Service" RedisConnectionPoolService. It says if configured it will use ssl
>> to establish a secure connection, if it's not configured it will not be
>> secure. And It is not configured.
>>
>> Please any help :-)
>>
>> kind regards
>> Jens M. Kofoed
>>
>>
