Redis supplies an official replication mechanism , and slave communicates
to master with RESP protocol, so a natural way to design the
rocketmq-redis-replicator component is simulating itself as a slave,
sending commands to master and receiving datas from master timely, and then
resending to rocketmq broker.
If you are not familiar with redis replication mechanism, please learn
this section first [1]. After that, I will illustrate some key points ahead.
1. To make slave start from the point where it left off when it reconnects,
slave and master should agree on a master runId and a replication offset.
Slave acknowledges this offset to master periodically. In other words，slave
may received duplicate commands. Along with, the rocketmq-redis-replicator
component may send duplicate messages too. A good way to minimize the
duplicate time window is reducing the "ack period" to a smaller one, such
as 100ms.
2. If slave keeps offline for some time, it’s easy to use up backlog whose
default value is just 1M, especially for a high-traffic redis instance.
Unfortunatelly,if slave replication offset has already been covered in
master backlog, a full synchronization will have to execute, which is
unacceptable for rocketmq-redis-replicator component as a large number of
messages will be sent out intensively.
3. When synchronizing from master fully, master will generate a new rdb
file(the rdb file format [2])，and slave will receive this file,store in
disk, and last apply to memory. This strategy makes slave reaches a
consistent state with master as soon as possible, and hardly fail. For
rocketmq-redis-replicator component, it’s also a good way to prevent
synchronizing initial rdb file from failure in halfway.
There already an open source project [3] which focuses on replicating
redis data, and provides api to handle data received [4]. The principal
thoughts are simulating itself as a slave , following official replication
procedure, communicating with master by RESP, and acking master with
replication offset. Base on this project to develop is a good idea,
meanwhile some aspects should also be enhanced and considered more robust.
Here is some points:
[High Available]
Keeping the replication component's high availability is not difficult
but important, not only for providing an uninterruptible service. If
component leaves off for some time, a unacceptable full synchronization may
be triggered.
It’s also easy to reach high availability, including adopting
master/slave module, using zookeeper to coordinate and switch master/slave,
storing data onto zookeeper to keep component stateless.
[Data Loss]
Generally, data loss should be tried best to avoid. The key point is
that slave only acks replication offset to master after sending command to
rocketmq broker successfully.
[Data Stale]
It also happened when slave reconnect. Consider case below:
`time1` `time2` `time3`
set k=a set k=b set k=c
If slave left off at time3, but the latest replication offset reported
to master is only at time1, when slave reconnected, it re-apply commands
“set k=b… set k=c”. In a small time window, “k” will equal the stale “b”
until “set k=c” command is applied. So the slave offline time shorter, the
better.
[Message Order]
Redis uses single thread model to keep command execute in order, because
of its high performance. Replicating data with a single thread in slave is
also fine, as it is also totally memory operation. But sending all data to
rocketmq in a global order is a good choose? Producer should have no
performance issue, but consumer may not be able to consume messages in
time, especially redis was in a high load.
Hashing “KEY” to different rocketmq queue is a good strategy. Guarantee
the same key operation route to a unique queue, to keep partial ordered,
and the downstream consumer could consume messages concurrently. Of course,
some dependency “KEY”s may need hash to a unique queue too. We should
supply configuration or api to support this individuation.
[Transaction]
Redis supports simple transaction. A transaction starts with a “MULTI”
command, and redis buffers latter commands and execute them until receiving
a “EXEC” command. But if one of the buffered commands executes fail, the
preceding executed commands won’t roll back, and the latter commands will
be discarded. So redis transaction could not guarantee atomic.
In rocketmq, it’s also impossible to gather multi messages consume
operation into a transaction. But the rocketmq-redis-replicator component
will just receive multi commands after redis server get a “EXEC” command.
>From this aspect, the “transaction semantic doesn’t strengthen or weaken
after this component resend messages.
[Avoid component switched to master]
In Sentinel or Redis Cluster, master crash could be detected by some
mechanism, and one slave will be switched to master. Master has all
information about its slaves, and the candidate slave is picked up
automatically. Obviously, rocketmq-redis-replicator component have no
ability to undertake master role. Configure this component to “read only”
slave is a good way to avoid the component switch to master.
[Support Redis Cluster]
No matter Redis Cluster, or previous Partition, a slave only keeps track
of one master. So the replication mechanism won’t change in single-node
redis instance or a redis cluster.
[1] : https://redis.io/topics/replication
[2]:
https://github.com/sripathikrishnan/redis-rdb-tools/wiki/Redis-RDB-Dump-File-Format
[3]: https://github.com/leonchen83/redis-replicator
[4]:
https://github.com/leonchen83/redis-replicator#31-replication-via-socket
2017-05-02 23:10 GMT+08:00 zhangke <zhangke_beij...@foxmail.com>:
> Sorry for the late reply. It's my fault for neglecting a design document,
> and I will provide it in recent days.
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "stevenschew";<stevensc...@apache.org>;
> *发送时间:* 2017年4月26日(星期三) 上午9:14
> *收件人:* "users"<us...@rocketmq.incubator.apache.org>;
> *主题:* Re: Proposal of Redis Data Replicator
>
> Hi, Rick, it is a good idea. To accommodate with many people involved in
> open source projects, is our consistent purpose.On the basis of the
> above, whether it can initially produce a detailed design documents. After
> the design document will be sent to the user list, we discuss the
> discussion. Maybe,
> prior to this, a specific project implementation plan needs to be
> developed and send it to the user list. That's all, thanks
>
> 2017-04-17 20:03 GMT+08:00 Rick Zhang <zhangke.huangs...@gmail.com>:
>
>> Hi users,devs,
>>
>> I am working on the component of redis data replicator, which
>> replicates datas from redis to RMQ for consuming in real time), and welcome
>> any volunteers to get involved :-)
>>
>> Before starting this work, I spent some time on searching if there
>> already have some good projects focus on this topic. Fortunately, such a
>> project [1] has been created and maintained for a year, and have got 79
>> stars. The principal thoughts are implementing redis replicator protocol,
>> and simulating itself as a slave to fetch master data in real time. In
>> addition, it could also keep track of aof file to get commands.
>>
>> After getting in touch with the author, I got the permission to use
>> this project in RMQ as a maven dependency, and meanwhile got more details.
>> The project has been deployed on some company's production environment ,
>> such as eastmoney.com with 150G memory redis instance.
>>
>> So I hope to collaborate with this open source project to accomplish
>> RMQ Redis Data Replicator, instead of developing a brand new one. Once
>> getting data from redis master is fast and stable, resending to RMQ will be
>> easy to complete.
>>
>>
>>
>> [1]: https://github.com/leonchen83/redis-replicator
>>
>
>
