Hi devs,
I'd like to update the state of RocketMQ Streaming Integration:
Now the task for RocketMQ-Flink integration is completed and patch is
available here: https://github.com/apache/rocketmq-externals/pull/45
Following is the brief changelog:
- RocketMQSource - The RocketMQSource is based on RocketMQ pull consumer
mode, and provides exactly once reliability guarantees when checkpoints are
enabled.
Otherwise, the source doesn't provide any reliability guarantees.
- RocketMQSink - The RocketMQSink provides at-least-once reliability
guarantees when checkpoints are enabled and
withBatchFlushOnCheckpoint(true) is set.
Otherwise, the sink reliability guarantees depends on rocketmq
producer's retry policy, for this case, the messages sending way is sync by
default,
but you can change it by invoking withAsync(true).
- KeyValueDeserializationSchema - The main API for deserializing topic
and tags is the
org.apache.rocketmq.flink.common.serialization.KeyValueDeserializationSchema
interface.
rocketmq-flink includes general purpose
KeyValueDeserializationSchema implementations
called SimpleKeyValueDeserializationSchema.
- KeyValueSerializationSchema - The main API for serializing topic and
tags is the
org.apache.rocketmq.flink.common.serialization.KeyValueSerializationSchema
interface.
rocketmq-flink includes general purpose KeyValueSerializationSchema
implementations
called SimpleKeyValueSerializationSchema.
- TopicSelector - The main API for selecting topic and tags is the
org.apache.rocketmq.flink.common.selector.TopicSelector interface.
rocketmq-flink includes general purpose TopicSelector implementations
called DefaultTopicSelector and SimpleTopicSelector.
- RocketMQFlinkExample - which receive messages from RocketMQ brokers
and send messages to broker after processing.
Any comments are welcome. And anybody who is also interested in these
tasks, please join me. Let's fight together.
Thanks,
Xin Wang
2018-01-19 21:47 GMT+08:00 yukon <[email protected]>:
> Cool, thanks for your outstanding contribution, to help RocketMQ community
> integrate with other streaming platforms.
>
> On Fri, Jan 19, 2018 at 7:52 PM, Xin Wang <[email protected]> wrote:
>
> > Hi devs,
> >
> > I have finished the improvements for integrating RocketMQ with Apache
> > Storm.
> > Main changes are as following:
> >
> > - Upgraded RocketMQ version to 4.2.0 which brings improvements and new
> > features like batch sending
> > - Imporved retry policy for RocketMQ consumer push mode to avoid data
> > loss in some scenes
> > - Batch sending supported for bolt and trident state
> > - Allow running several consumer instances in one process, that is to
> > say, different topics in one worker is possible
> >
> > PR: https://github.com/apache/storm/pull/2518
> >
> > And I submit the `RocketMQ-Serializer` patch several days ago. This
> module
> > includes several serialization formats, especially Apache Avro which I
> > stated before[1].
> >
> > - Raw String
> > - JSON
> > - Avro Generic
> > - Avro Specified
> >
> > PR: https://github.com/apache/rocketmq-externals/pull/42
> >
> > Any comments for these PRs are welcome. BTW here is the state of rocketmq
> > streaming integration:
> > Module Status
> > * RocketMQ-Storm Patch Available
> > * RocketMQ-Spark To Refactor
> > * RocketMQ-Flink Patch Available Soon
> > * RocketMQ-Serializer Patch Available
> >
> > Anybody who is also interested in these tasks, please join me. Let's
> fight
> > together.
> >
> >
> > [1] https://issues.apache.org/jira/browse/ROCKETMQ-157
> >
> >
> > Thanks,
> >
> > Xin Wang (vesense)
> >
>
--
Thanks,
Xin
