The Bahir-Flink stuff is fairly new - the first release has not yet happened. Robert is pushing that, will probably happen once he is back.
On Tue, Sep 27, 2016 at 3:31 PM, shijinkui <shijin...@huawei.com> wrote: > It's nice. Will present flink source connector be pushed to bahir-flink? > I can add netty-source to bahir-flink. > > Maven repository have no bahir-flink's. > https://mvnrepository.com/artifact/org.apache.bahir > > -----邮件原件----- > 发件人: Greg Hogan [mailto:c...@greghogan.com] > 发送时间: 2016年9月27日 20:58 > 收件人: dev@flink.apache.org > 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support > > Apache Bahir's website only suggests support for additional frameworks, > but there is a Flink repository at > https://github.com/apache/bahir-flink > > On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <shijin...@huawei.com> wrote: > > > Hey, Stephan Ewen > > > > 1. bahir's target is spark. The contributer are rxin, srowen, tdas, > > mateiz and so on. > > If we want bahir used by flink, we can suggest bahir provide > > streaming connecter interface, such as store(), start(), stop(), > > restart(), receiving(Any)... > > Then same streaming connector can be implemented by spark and > > flink. But I think this is impossible, as bahir depend spark-streaming > > and spark sql. > > 2. About connector fault tolerance. Bahir's mqtt and akka connector > > are themselves' storage. But netty have no persist data feature. > > I think we can append data to a ringbuffer. When SourceContext > > collect() throw error, then write message to inform client to stop > > send message. When flink SourceContext is normal, then write the > > ringbuffer data to flink, inform client to go on. > > Because pushing mode is hard to control the flow throughput, > > the upstream client can. > > This netty connector's purpose is end-to-end streaming, > > minimum time delay. > > 3. Later on, We can provide http protocol ability, put, post, get, > > even websocket or jersey restful based on netty. > > > > -----邮件原件----- > > 发件人: Stephan Ewen [mailto:se...@apache.org] > > 发送时间: 2016年9月27日 16:54 > > 收件人: dev@flink.apache.org > > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support > > > > I think that could be an interesting source. Two quick questions to > > move forward > > > > - To keep the Flink code base from becoming too big (hard to > > maintain and > > test) we started working with Apache Bahir as a project dedicated to > > streaming connectors. Would that be a good target for the connector? > > > > - What are your thoughts on fault tolerance for that connector? > > > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <shijin...@huawei.com> wrote: > > > > > Hi, all > > > > > > 1. In order to support end-to-end pushed source, I create > FLINK-4630< > > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know > > > whether is this idea worth? > > > > > > ------------------- > > > When source stream get start, listen a provided tcp port, receive > > > stream data from user data source. > > > This netty tcp source is keepping alive and end-to-end, that is from > > > business system to flink worker directly. > > > > > > user app push -----> netty server source of Flink > > > > > > describe the source in detail below: > > > > > > 1. source run as a netty tcp server > > > 2. user provide a tcp port, if the port is in used, increace the > port > > > number between 1024 to 65535. Source can parallel. > > > 3. callback the provided url to report the real port to listen > > > 4. user push streaming data to netty server, then collect the data > to > > > flink > > > > > > > > > Thanks > > > > > > Jinkui Shi > > > > > > > > >
