I found the bug. - Reported as https://issues.apache.org/jira/browse/NIFI-9019
If I select database index 0, everything works fine. My process which is
not working is using a database index of 3.
So changing my small test flow (which is working) from database index 0 ->
3 produce the same errors in the log and stop working.
I don't know what should make this stop working, but I works fine in
version 1.13.2 but not in version 1.14.0
Create a RedisConnectionPoolService with Database Index = 0
Test attach flow: everything should work
Change Database Index = 3
tail -f the nifi-app.log file and start the the flow again.
kind Regards
Jens M. Kofoed
Den fre. 6. aug. 2021 kl. 11.31 skrev Jens M. Kofoed <jmkofoed....@gmail.com
>:
> Some more digging
>
> I restarted my Redis cluster. Created a new RedisConnectionPoolService and
> a new RedisDistributedMapCacheClientService.
> I created a small test flow: Genereate flowfile -> PutDistributedMapCache
> -> AttributeUpdate
> No problems, it works but...
>
> I went back to one the other flows where I use a DetectDuplicate process
> and I got the following lines in the log
>
> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13]
> o.a.n.r.s.RedisConnectionPoolService
> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b]
> Connecting to Redis in sentinel mode...
> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13]
> o.a.n.r.s.RedisConnectionPoolService
> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis
> master = mymaster
> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13]
> o.a.n.r.s.RedisConnectionPoolService
> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis
> sentinel at redis-01:26379
> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13]
> o.a.n.r.s.RedisConnectionPoolService
> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis
> sentinel at redis-02:26379
> 2021-08-06 11:18:21,123 INFO [Timer-Driven Process Thread-13]
> o.a.n.r.s.RedisConnectionPoolService
> RedisConnectionPoolService[id=160a1056-017b-1000-0000-00005533ce4b] Redis
> sentinel at redis-03:26379
> 2021-08-06 11:18:21,124 INFO [Timer-Driven Process Thread-13]
> redis.clients.jedis.JedisSentinelPool Trying to find master from available
> Sentinels...
> 2021-08-06 11:18:21,124 DEBUG [Timer-Driven Process Thread-13]
> redis.clients.jedis.JedisSentinelPool Connecting to Sentinel redis-01:26379
> 2021-08-06 11:18:21,124 WARN [NiFi Web Server-4964]
> org.apache.nifi.audit.ProcessorAuditor Unable to record actions: null
> 2021-08-06 11:18:21,214 INFO [Flow Service Tasks Thread-1]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@33ab3784 // Another save
> pending = false
> 2021-08-06 11:18:31,134 WARN [Timer-Driven Process Thread-13]
> redis.clients.jedis.JedisSentinelPool Cannot get master address from
> sentinel running @ i-redis-01:26379. Reason: {}. Trying next one.
> redis.clients.jedis.exceptions.JedisConnectionException:
> javax.net.ssl.SSLException: Read timed out
> at redis.clients.jedis.Connection.flush(Connection.java:342)
>
> So it connects to me Redis Sentinel cluster, but later it somehow trying
> to use SSL or what???
>
> kind regards
> Jens M. Kofoed
>
> Den fre. 6. aug. 2021 kl. 08.07 skrev Jens M. Kofoed <
> jmkofoed....@gmail.com>:
>
>> Hi
>>
>> I found another WARN in my log, which is about SSLException. However I
>> did NOT specified the new property: SSL Context Service
>> From the documentation: If specified, this service will be used to create
>> an SSL Context that will be used to secure communications; if not
>> specified, communications will not be secure
>>
>> 2021-08-06 07:47:30,166 WARN [Timer-Driven Process Thread-12]
>> redis.clients.jedis.JedisSentinelPool Cannot get master address from
>> sentinel running @ redis-03:26379. Reason: {}. Trying next one.
>> redis.clients.jedis.exceptions.JedisConnectionException:
>> javax.net.ssl.SSLException: Read timed out
>> at redis.clients.jedis.Connection.flush(Connection.java:342)
>> at
>> redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:269)
>> at redis.clients.jedis.BinaryJedis.select(BinaryJedis.java:787)
>> at
>> redis.clients.jedis.BinaryJedis.initializeFromClientConfig(BinaryJedis.java:99)
>> at redis.clients.jedis.BinaryJedis.<init>(BinaryJedis.java:82)
>> at redis.clients.jedis.Jedis.<init>(Jedis.java:54)
>> at
>> redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:251)
>> at
>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205)
>> at
>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193)
>> at
>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411)
>> at
>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393)
>> at
>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348)
>> at
>> org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342)
>> at
>> org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92)
>> at sun.reflect.GeneratedMethodAccessor1212.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
>> at
>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
>> at com.sun.proxy.$Proxy181.getConnection(Unknown Source)
>> at
>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375)
>> at
>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144)
>> at sun.reflect.GeneratedMethodAccessor1211.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
>> at
>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
>> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source)
>> at
>> org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183)
>> at
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
>> at
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
>> at
>> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: javax.net.ssl.SSLException: Read timed out
>> at sun.security.ssl.Alert.createSSLException(Alert.java:127)
>> at
>> sun.security.ssl.TransportContext.fatal(TransportContext.java:324)
>> at
>> sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
>> at
>> sun.security.ssl.TransportContext.fatal(TransportContext.java:262)
>> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:138)
>> at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1383)
>> at
>> sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1291)
>> at
>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435)
>> at
>> sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:804)
>> at
>> sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:73)
>> at
>> sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1166)
>> at
>> redis.clients.jedis.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52)
>> at
>> redis.clients.jedis.util.RedisOutputStream.flush(RedisOutputStream.java:133)
>> at redis.clients.jedis.Connection.flush(Connection.java:339)
>> ... 40 common frames omitted
>> Caused by: java.net.SocketTimeoutException: Read timed out
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at
>> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>> at java.net.SocketInputStream.read(SocketInputStream.java:171)
>> at java.net.SocketInputStream.read(SocketInputStream.java:141)
>> at
>> sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
>> at
>> sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165)
>> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109)
>> ... 49 common frames omitted
>>
>> Kind regards
>> Jens M. Kofoed
>>
>> Den fre. 6. aug. 2021 kl. 06.35 skrev Jens M. Kofoed <
>> jmkofoed....@gmail.com>:
>>
>>> Dear Develovper
>>>
>>> I'm using many RedisDistributedMapCacheClientService and
>>> some RedisConnectionPoolService which connect to my 3 node cluster
>>> with Redis sentinels.
>>> After I upgraded one of my NiFi servers to 1.14.0 the processors which
>>> is using the Redis start having issues. Looking in the logs I got the
>>> following errors/WARN.
>>>
>>> 2021-08-05 13:24:12,556 WARN [Timer-Driven Process Thread-3]
>>> o.a.n.controller.tasks.ConnectableTask Administratively Yielding
>>> DetectDuplicate[id=feb1f2ff-4f9c-31d7-87c8-e225a80ef03f] due to uncaught
>>> Exception: redis.clients.jedis.exceptions.JedisConnectionException: All
>>> sentinels down, cannot determine where is mymaster master is running...
>>> redis.clients.jedis.exceptions.JedisConnectionException: All sentinels
>>> down, cannot determine where is mymaster master is running...
>>> at
>>> redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:281)
>>> at
>>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205)
>>> at
>>> redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:193)
>>> at
>>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:411)
>>> at
>>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:393)
>>> at
>>> org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348)
>>> at
>>> org.apache.nifi.redis.util.RedisUtils.createConnectionFactory(RedisUtils.java:342)
>>> at
>>> org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(RedisConnectionPoolService.java:92)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at
>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
>>> at
>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
>>> at com.sun.proxy.$Proxy181.getConnection(Unknown Source)
>>> at
>>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConnection(RedisDistributedMapCacheClientService.java:375)
>>> at
>>> org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.getAndPutIfAbsent(RedisDistributedMapCacheClientService.java:144)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at
>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
>>> at
>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
>>> at com.sun.proxy.$Proxy179.getAndPutIfAbsent(Unknown Source)
>>> at
>>> org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183)
>>> at
>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>> at
>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
>>> at
>>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
>>> at
>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>> I know the WARN says: All sentinels down, cannot determine where is
>>> mymaster master is running... BUT It is not. The Sentinel master is
>>> running, I have a 2nd NIFI server running with v. 1.13.2 which is using the
>>> same Redis cluster and that one is still working. If I change the server
>>> (which has issues in v. 1.14.0) back to 1.13.2 there are no errors. I also
>>> tried to change one of the other servers to v. 1.14.0 and it have the same
>>> issue.
>>>
>>> A new feature in v.1.14.0 is the property "SSL Context
>>> Service" RedisConnectionPoolService. It says if configured it will use ssl
>>> to establish a secure connection, if it's not configured it will not be
>>> secure. And It is not configured.
>>>
>>> Please any help :-)
>>>
>>> kind regards
>>> Jens M. Kofoed
>>>
>>>
RedisTest.json
Description: application/json
