Re: [Discussion] Flink Pulsar Connector

[Henry Saputra]

Here is the link to Apache Flink JIRA issue for this:

https://issues.apache.org/jira/browse/FLINK-9168

- Henry

On Fri, Apr 20, 2018 at 12:08 AM, Sijie Guo  wrote:

> Hi Flinkers,
>
> As discussed with @tzulitai at apache/flink#5845
> , I am starting a discussion
> thread about contributing flink pulsar connectors (including both source
> and sink connectors) from pulsar community to flink project. We'd like to
> see what are people's thoughts about this and how we can proceed for this.
>
> For people who doesn't know about Apache Pulsar, here are some background:
>
> ---
>
> Apache Pulsar (incubating)  is a
> distributed pub/sub messaging system, which provides very flexible
> messaging model - unifying traditional queuing (e.g. SQS, rabbitmq) and
> high-performance streaming (e.g. Kinesis, Kafka) into one pub/sub messaging
> model + api. It is backed by a scalable segment/log storage Apache
> BookKeeper, which provide unbounded stream storage for Pulsar. Because of
> its segment-centric architecture design, Pulsar provides compelling
> unbounded streaming data storage. It is good for both streaming and batch
> processing, which I believe it fits very well into Flink's data processing
> model. Besides that, pulsar has a lot of advanced features going on its
> upcoming 2.0 release, including built-in schema registry, topic compaction,
> regex subscription, and tiered storage
>  Tiered-storage-for-Pulsar-topics>
>  ...
>
> Pulsar was developed by Yahoo since 2012-ish and has been running on
> production for 4+ years, over 10+ data centers and processing/delivering
> billions of messages per day. It was open sourced at 2016. Since it is open
> sourced, it has been adopted by various companies. Nowadays, the pulsar
> slack channel discussion is very active and fast-growing. The community
> currently has about 15 committers.
>
> ---
>
> I happened to work with ZongYang (who is also a pulsar contributor) on
> developing pulsar connectors for flink to satisfy pulsar users requests. We
> would like to contribute the connector work to flink and continue the
> collaboration between flink and pulsar communities. From pulsar community
> perspective, we are also very committed to developing pulsar's ecosystem,
> and willing and dedicated to developing/maintaining flink pulsar
> connectors.
>
> Hope this email thread give you guys enough background of pulsar and clear
> some of the concerns that @tzulitai raised in the jira ticket / pull
> request. Looking forward to any feedback from pulsar community and deep
> collaboration between flink and pulsar community.
>
> Also /cc pulsar dev mailing list (d...@pulsar.incubator.apache.org). If
> there are any questions, pulsar devs can also help to answer.
>
> Thanks,
> Sijie
>

[jira] [Created] (FLINK-9232) Add harness test for AggregationCodeGenerator

[Rong Rong (JIRA)]

Rong Rong created FLINK-9232:


 Summary: Add harness test for AggregationCodeGenerator 
 Key: FLINK-9232
 URL: https://issues.apache.org/jira/browse/FLINK-9232
 Project: Flink
  Issue Type: Sub-task
Reporter: Rong Rong


Instead of relying on ITCase to cover the codegen result. We should have direct 
test against that, for example using Harness test framework.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9231) Enable SO_REUSEADDR on listen sockets

[Ted Yu (JIRA)]

Ted Yu created FLINK-9231:
-

 Summary: Enable SO_REUSEADDR on listen sockets
 Key: FLINK-9231
 URL: https://issues.apache.org/jira/browse/FLINK-9231
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


This allows sockets to be bound even if there are sockets
from a previous application that are still pending closure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)