Re: The State of RocketMQ Streaming Integration

[Xin Wang]

Thanks very much. Glad to see that more users use and participate in the
bigdata eco-system connectors.

- Xin

2018-04-23 19:59 GMT+08:00 Von Gosling :

> HI Xing,
>
>
> Cool, some guys have looking for me to ask for help in the RocketMQ Data
> Connector, I am glad to introduce your works to them. Let’s put forward it
> together :-)
>
> BTW, I will merge some updates of the wiki which you have provided.
>
> Best Regards,
> Von Gosling
>
> > 在 2018年4月20日，14:47，Xin Wang  写道：
> >
> > Hi folks,
> >
> > Now the improvements for RocketMQ-Storm has been merged into master.
> >
> > Here is the latest state of rocketmq streaming integration:
> >ModuleStatus
> > * RocketMQ-StormAvailable
> > * RocketMQ-SparkAvailable, improvements patch to be merged.
> > * RocketMQ-Flink Available
> > * RocketMQ-Serializer(including JSON, Apache Avro)Available, to be
> merged.
> >
> > Some new modules on the road:
> > * RocketMQ-SQL  https://issues.apache.org/jira/browse/ROCKETMQ-357 <
> https://issues.apache.org/jira/browse/ROCKETMQ-357>
> > * RocketMQ-Beam(Beam IO)
> >
> >
> > Recently, I'm working on the RocketMQ SQL. Hope the initial version can
> be released ASAP. Anybody who is interested in please join me.
> >
> >
> > Thanks,
> > Xin Wang
> >
>
>


-- 
Thanks,
Xin

Re: The State of RocketMQ Streaming Integration

[Von Gosling]

HI Xing,


Cool, some guys have looking for me to ask for help in the RocketMQ Data 
Connector, I am glad to introduce your works to them. Let’s put forward it 
together :-)

BTW, I will merge some updates of the wiki which you have provided.

Best Regards,
Von Gosling

> 在 2018年4月20日，14:47，Xin Wang  写道：
> 
> Hi folks,
> 
> Now the improvements for RocketMQ-Storm has been merged into master.
> 
> Here is the latest state of rocketmq streaming integration:
>ModuleStatus
> * RocketMQ-StormAvailable
> * RocketMQ-SparkAvailable, improvements patch to be merged.
> * RocketMQ-Flink Available
> * RocketMQ-Serializer(including JSON, Apache Avro)Available, to be merged.
> 
> Some new modules on the road:
> * RocketMQ-SQL  https://issues.apache.org/jira/browse/ROCKETMQ-357 
> 
> * RocketMQ-Beam(Beam IO)
> 
> 
> Recently, I'm working on the RocketMQ SQL. Hope the initial version can be 
> released ASAP. Anybody who is interested in please join me.
> 
> 
> Thanks,
> Xin Wang
> 

Re: The State of RocketMQ Streaming Integration

[Xin Wang]

Hi folks,

Now the improvements for RocketMQ-Storm has been merged into master.

Here is the latest state of rocketmq streaming integration:
   ModuleStatus
* RocketMQ-Storm*Available*
* RocketMQ-Spark*Available, improvements patch to be merged.*
* RocketMQ-Flink *Available*
* RocketMQ-Serializer(including JSON, Apache Avro)*Available, to be
merged.*

Some new modules on the road:
* RocketMQ-SQL  https://issues.apache.org/jira/browse/ROCKETMQ-357
* RocketMQ-Beam(Beam IO)


Recently, I'm working on the RocketMQ *SQL*. Hope the initial version can
be released ASAP. Anybody who is interested in please join me.


Thanks,
Xin Wang

Re: The State of RocketMQ Streaming Integration

[yukon]

Thanks, Xin!

These plugins have greatly enriched the RocketMQ community, hope we could
get more feedback from the community about these plugins.

Regards,
yukon


On Mon, Mar 26, 2018 at 10:44 AM, Xin Wang  wrote:

> Hi folks,
>
> I'm glad to say that the RocketMQ-Flink module has been merged into master.
> https://github.com/apache/rocketmq-externals/tree/master/rocketmq-flink
>
> Here is the state of rocketmq streaming integration:
>ModuleStatus
> * RocketMQ-Storm*Available*
> * RocketMQ-Spark*Available*, *The improvements will be available soon*.
> * RocketMQ-Flink *Available*
> * RocketMQ-Serializer(including JSON, Apache Avro)*Available*, *Not
> merged yet*.
>
> And, some new modules on the road:
> * RocketMQ-SQL  https://issues.apache.org/jira/browse/ROCKETMQ-357
> * RocketMQ-Beam(Beam IO)
>
>
> Anybody who is interested in these tasks too, please join me. Let's fight
> together.
>
>
> Thanks,
> Xin Wang
>
>
>
> 2018-01-27 19:35 GMT+08:00 Xin Wang :
>
> > Hi devs,
> >
> > I'd like to update the state of RocketMQ Streaming Integration:
> > Now the task for RocketMQ-Flink  integration is completed and  patch is
> > available here: https://github.com/apache/rocketmq-externals/pull/45
> > Following is the brief changelog:
> >
> >- RocketMQSource - The RocketMQSource is based on RocketMQ pull
> >consumer mode, and provides exactly once reliability guarantees when
> >checkpoints are enabled.
> >Otherwise, the source doesn't provide any reliability guarantees.
> >- RocketMQSink - The RocketMQSink provides at-least-once reliability
> >guarantees when checkpoints are enabled and withBatchFlushOnCheckpoint
> >(true) is set.
> >Otherwise, the sink reliability guarantees depends on rocketmq
> >producer's retry policy, for this case, the messages sending way is
> sync by
> >default,
> >but you can change it by invoking withAsync(true).
> >- KeyValueDeserializationSchema - The main API for deserializing topic
> >and tags is the org.apache.rocketmq.flink.common.serialization.
> >KeyValueDeserializationSchema interface.
> >rocketmq-flink includes general purpose KeyValueDeserializationSchema
> implementations
> >called SimpleKeyValueDeserializationSchema.
> >- KeyValueSerializationSchema - The main API for serializing topic and
> >tags is the org.apache.rocketmq.flink.common.serialization.
> >KeyValueSerializationSchema interface.
> >rocketmq-flink includes general purpose KeyValueSerializationSchema
> implementations
> >called SimpleKeyValueSerializationSchema.
> >- TopicSelector - The main API for selecting topic and tags is the
> >org.apache.rocketmq.flink.common.selector.TopicSelector interface.
> >rocketmq-flink includes general purpose TopicSelector implementations
> >called DefaultTopicSelector and SimpleTopicSelector.
> >- RocketMQFlinkExample - which receive messages from RocketMQ brokers
> >and send messages to broker after processing.
> >
> > Any comments are welcome. And anybody who is also interested in these
> > tasks, please join me. Let's fight together.
> >
> > Thanks,
> > Xin Wang
> >
> >
> > 2018-01-19 21:47 GMT+08:00 yukon :
> >
> >> Cool, thanks for your outstanding contribution, to help RocketMQ
> community
> >> integrate with other streaming platforms.
> >>
> >> On Fri, Jan 19, 2018 at 7:52 PM, Xin Wang 
> wrote:
> >>
> >> > Hi devs,
> >> >
> >> > I have finished the improvements for integrating RocketMQ with Apache
> >> > Storm.
> >> > Main changes are as following:
> >> >
> >> >- Upgraded RocketMQ version to 4.2.0 which brings improvements and
> >> new
> >> >features like batch sending
> >> >- Imporved retry policy for RocketMQ consumer push mode to avoid
> data
> >> >loss in some scenes
> >> >- Batch sending supported for bolt and trident state
> >> >- Allow running several consumer instances in one process, that is
> to
> >> >say, different topics in one worker is possible
> >> >
> >> > PR: https://github.com/apache/storm/pull/2518
> >> >
> >> > And I submit the `RocketMQ-Serializer` patch several days ago. This
> >> module
> >> > includes several serialization formats, especially Apache Avro which I
> >> > stated before[1].
> >> >
> >> >- Raw String
> >> >- JSON
> >> >- Avro Generic
> >> >- Avro Specified
> >> >
> >> > PR: https://github.com/apache/rocketmq-externals/pull/42
> >> >
> >> > Any comments for these PRs are welcome. BTW here is the state of
> >> rocketmq
> >> > streaming integration:
> >> >ModuleStatus
> >> > * RocketMQ-StormPatch Available
> >> > * RocketMQ-SparkTo Refactor
> >> > * RocketMQ-Flink Patch Available Soon
> >> > * RocketMQ-Serializer Patch Available
> >> >
> >> > Anybody who is also interested in these tasks, please join me. Let's
> >> fight
> >> > together.
> >> >
> >> >
> >>