On 15 Mar 06:40, Theo Buehler wrote:
> > Hi Theo,
> >
> > just "back ported" redis to 6.6 and it works like a charm on amd64 (except
> > sentinel which segfaults - but it did also on 4.0.14).
> >
>
> Thanks. Yes, there should be no major problems backporting this.
>
> Could you share a reproducer for this sentinel segfault so I can look
> into it?
>
Hi Theo,
I tried the following setup:
1 master + 3 slaves. On all machines running sentinel with the following
(default) config:
######## /etc/redis/sentinel.conf
protected-mode no
port 26379
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster A.B.C.23 6379 2
dir "/tmp"
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 10000
#######
I run sentinel as user _redis so the owner of the config is _redis.
Start sentinel on all nodes:
doas -u _redis /usr/local/sbin/redis-sentinel /etc/redis/sentinel.conf
--sentinel
21836:X 15 Mar 2020 17:09:59.806 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
21836:X 15 Mar 2020 17:09:59.807 # Redis version=5.0.8, bits=64,
commit=00000000, modified=0, pid=21836, just started
21836:X 15 Mar 2020 17:09:59.807 # Configuration loaded
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 5.0.8 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in sentinel mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 26379
| `-._ `._ / _.-' | PID: 21836
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
21836:X 15 Mar 2020 17:09:59.811 # Sentinel ID is
7c56c258f3b6d91a276d66d37edb4a7d34fd46db
21836:X 15 Mar 2020 17:09:59.811 # +monitor master mymaster A.B.C.23 6379
quorum 2
21836:X 15 Mar 2020 17:10:04.874 # +sdown slave A.B.C.52:6379 A.B.C.52 6379 @
mymaster A.B.C.D 6379
21836:X 15 Mar 2020 17:10:04.874 # +sdown slave A.B.C.51:6379 A.B.C.51 6379 @
mymaster A.B.C.D 6379
21836:X 15 Mar 2020 17:10:55.617 # +sdown sentinel
fcb4b40293a180b56f4e00c9c0ac6c719f8b3251 A.B.C.51 26379 @ mymaster A.B.C.23 6379
21836:X 15 Mar 2020 17:11:58.393 # +sdown sentinel
1891b2bf811eae3ccf2e9dee9ad71f234148f6d7 A.B.C.24 26379 @ mymaster A.B.C.23 6379
21836:X 15 Mar 2020 17:12:02.303 # -sdown sentinel
fcb4b40293a180b56f4e00c9c0ac6c719f8b3251 A.B.C.51 26379 @ mymaster A.B.C.23 6379
21836:X 15 Mar 2020 17:12:22.703 # -sdown sentinel
1891b2bf811eae3ccf2e9dee9ad71f234148f6d7 A.B.C.24 26379 @ mymaster A.B.C.23 6379
21836:X 15 Mar 2020 17:12:26.459 # +sdown sentinel
fcb4b40293a180b56f4e00c9c0ac6c719f8b3251 A.B.C.51 26379 @ mymaster A.B.C.23 6379
21836:X 15 Mar 2020 17:12:35.003 # -sdown sentinel
fcb4b40293a180b56f4e00c9c0ac6c719f8b3251 A.B.C.51 26379 @ mymaster A.B.C.23 6379
Segmentation fault
This happens after some time on all 4 nodes.
Kdump shows this:
94624 redis-server RET write 14/0xe
94624 redis-server CALL kevent(5,0x7f7ffffbddb0,1,0,0,0)
94624 redis-server STRU struct kevent { ident=12, filter=EVFILT_WRITE,
flags=0x2<EV_DELETE>, fflags=0<>, data=0, udata=0x0 }
94624 redis-server RET kevent 0
94624 redis-server CALL gettimeofday(0x7f7ffffbde10,0)
94624 redis-server STRU struct timeval { 1584292747<"Mar 15 17:19:07
2020">.390804 }
94624 redis-server RET gettimeofday 0
94624 redis-server CALL gettimeofday(0x7f7ffffbde60,0)
94624 redis-server STRU struct timeval { 1584292747<"Mar 15 17:19:07
2020">.390820 }
94624 redis-server RET gettimeofday 0
94624 redis-server CALL gettimeofday(0x7f7ffffbddd8,0)
94624 redis-server STRU struct timeval { 1584292747<"Mar 15 17:19:07
2020">.390837 }
94624 redis-server RET gettimeofday 0
94624 redis-server CALL read(3,0x7f7ffffbde4f,0x1)
94624 redis-server RET read -1 errno 35 Resource temporarily unavailable
94624 redis-server CALL gettimeofday(0x7f7ffffbde60,0)
94624 redis-server STRU struct timeval { 1584292747<"Mar 15 17:19:07
2020">.390868 }
94624 redis-server RET gettimeofday 0
94624 redis-server CALL kevent(5,0,0,0x1b3c3e9e0008,224,0x7f7ffffbde60)
94624 redis-server STRU struct timespec { 0.058000000 }
94624 redis-server STRU struct kevent { ident=8, filter=EVFILT_READ,
flags=0x1<EV_ADD>, fflags=0<>, data=2896, udata=0x0 }
94624 redis-server RET kevent 1
94624 redis-server CALL read(8,0x7f7ffffb9de0,0x4000)
94624 redis-server GIO fd 8 read 3338 bytes
"$3329\r
# Server\r
redis_version:5.0.8\r
redis_git_sha1:00000000\r
redis_git_dirty:0\r
redis_build_id:fa7e9cf2d63d15f4\r
redis_mode:standalone\r
os:OpenBSD 6.6 amd64\r
arch_bits:64\r
multiplexing_api:kqueue\r
atomicvar_api:atomic-builtin\r
gcc_version:4.2.1\r
process_id:51797\r
run_id:af018b4eb67b48a5d994ec0dbfbad652dbd94613\r
tcp_port:6379\r
uptime_in_seconds:57557\r
uptime_in_days:0\r
hz:10\r
configured_hz:10\r
lru_clock:7234443\r
executable:/usr/local/sbin/redis-server\r
config_file:/etc/redis/redis.conf\r
\r
# Clients\r
connected_clients:2\r
client_recent_max_input_buffer:2\r
client_recent_max_output_buffer:0\r
blocked_clients:0\r
\r
# Memory\r
used_memory:1484880\r
used_memory_human:1.42M\r
used_memory_rss:1451088\r
used_memory_rss_human:1.38M\r
used_memory_peak:1687496\r
used_memory_peak_human:1.61M\r
used_memory_peak_perc:87.99%\r
used_memory_overhead:1481914\r
used_memory_startup:349792\r
used_memory_dataset:2966\r
used_memory_dataset_perc:0.26%\r
allocator_allocated:1451088\r
allocator_active:1413200\r
allocator_resident:1413200\r
total_system_memory:0\r
total_system_memory_human:0B\r
used_memory_lua:37888\r
used_memory_lua_human:37.00K\r
used_memory_scripts:0\r
used_memory_scripts_human:0B\r
number_of_cached_scripts:0\r
maxmemory:0\r
maxmemory_human:0B\r
maxmemory_policy:noeviction\r
allocator_frag_ratio:0.97\r
allocator_frag_bytes:18446744073709513728\r
allocator_rss_ratio:1.00\r
allocator_rss_bytes:0\r
rss_overhead_ratio:1.03\r
rss_overhead_bytes:37888\r
mem_fragmentation_ratio:1.00\r
mem_fragmentation_bytes:0\r
mem_not_counted_for_evict:0\r
mem_replication_backlog:1048584\r
mem_clients_slaves:16922\r
mem_clients_normal:66616\r
mem_aof_buffer:0\r
mem_allocator:libc\r
active_defrag_running:0\r
lazyfree_pending_objects:0\r
\r
# Persistence\r
loading:0\r
rdb_changes_since_last_save:0\r
rdb_bgsave_in_progress:0\r
rdb_last_save_time:1584235190\r
rdb_last_bgsave_status:ok\r
rdb_last_bgsave_time_sec:-1\r
rdb_current_bgsave_time_sec:-1\r
rdb_last_cow_size:0\r
aof_enabled:0\r
aof_rewrite_in_progress:0\r
aof_rewrite_scheduled:0\r
aof_last_rewrite_time_sec:-1\r
aof_current_rewrite_time_sec:-1\r
aof_last_bgrewrite_status:ok\r
aof_last_write_status:ok\r
aof_last_cow_size:0\r
\r
# Stats\r
total_connections_received:27\r
total_commands_processed:53496\r
instantaneous_ops_per_sec:1\r
total_net_input_bytes:2046405\r
total_net_output_bytes:584703\r
instantaneous_input_kbps:0.04\r
instantaneous_output_kbps:0.01\r
rejected_connections:0\r
sync_full:0\r
sync_partial_ok:3\r
sync_partial_err:0\r
expired_keys:0\r
expired_stale_perc:0.00\r
expired_time_cap_reached_count:0\r
evicted_keys:0\r
keyspace_hits:0\r
keyspace_misses:0\r
pubsub_channels:1\r
pubsub_patterns:0\r
latest_fork_usec:0\r
migrate_cached_sockets:0\r
slave_expires_tracked_keys:0\r
active_defrag_hits:0\r
active_defrag_misses:0\r
active_defrag_key_hits:0\r
active_defrag_key_misses:0\r
\r
# Replication\r
role:master\r
connected_slaves:1\r
slave0:ip=A.B.C.24,port=6379,state=online,offset=127698,lag=1\r
master_replid:2db1b4c0ae966ac5703dc02a70b10aa116229f00\r
master_replid2:b8858c4518db5ff25a6850a54b678183cbea74b2\r
master_repl_offset:127712\r
second_repl_offset:74663\r
repl_backlog_active:1\r
repl_backlog_size:1048576\r
repl_backlog_first_byte_offset:3137\r
repl_backlog_histlen:124576\r
\r
# CPU\r
used_cpu_sys:2.590000\r
used_cpu_user:4.570000\r
used_cpu_sys_children:0.000000\r
used_cpu_user_children:0.000000\r
\r
# Cluster\r
cluster_enabled:0\r
\r
# Keyspace\r
\r
"
94624 redis-server RET read 3338/0xd0a
94624 redis-server PSIG SIGSEGV SIG_DFL code SEGV_MAPERR<1>
addr=0x1b3b8673f000 trapno=6
94624 redis-server NAMI "redis-server.core"
###############
gdb /usr/local/sbin/redis-server /tmp/redis-server.core
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-unknown-openbsd6.6"...(no debugging symbols
found)
Core was generated by `redis-server'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/lib/libpthread.so.26.1...done.
Loaded symbols for /usr/lib/libpthread.so.26.1
Loaded symbols for /usr/local/sbin/redis-server
Reading symbols from /usr/lib/libm.so.10.1...done.
Loaded symbols for /usr/lib/libm.so.10.1
Symbols already loaded for /usr/lib/libpthread.so.26.1
Reading symbols from /usr/local/lib/liblua5.1.so.5.1...done.
Loaded symbols for /usr/local/lib/liblua5.1.so.5.1
Reading symbols from /usr/lib/libc.so.95.1...done.
Loaded symbols for /usr/lib/libc.so.95.1
Reading symbols from /usr/libexec/ld.so...Error while reading shared library
symbols:
Dwarf Error: wrong version in compilation unit header (is 4, should be 2) [in
module /usr/libexec/ld.so]
#0 0x0000127e6e9c4ff8 in SHA1Final () from /usr/local/sbin/redis-server
###############
Anything else I could help with?
With kind regards
Uwe
--
