[
https://issues.apache.org/jira/browse/FLINK-6085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939209#comment-15939209
]
Chen Qin edited comment on FLINK-6085 at 3/24/17 8:32 PM:
----------------------------------------------------------
I would like to see if we can agree on high level first. Service is primarily
rpc interaction with horizontal scalability and latency requirements.
Current way of bridge service with streaming pipeline via distributed Queue
provides benefit of failure resilience and topic reuse at cost of extra
hardware/software and latency, also no callback support.
[~till.rohrmann]
updates
Briefly chatted offline with Maxim, it seems a bit hard to work around
distributed queue consider pipeline can restart and offset rewind anytime, loss
of insertion events is not acceptable(query might be fine but seems flink
already address this issue with query able states)
To echo Till's comments, yes, custom code could track those requests. Future
question is if we can have a specific sink implementation which can reroute
results to a specific rpc hosts (e.g http response or callback).
was (Author: foxss):
I would like to see if we can agree on high level first. Service is primarily
rpc interaction with horizontal scalability and latency requirements.
Current way of bridge service with streaming pipeline via distributed Queue
provides benefit of failure resilience and topic reuse at cost of extra
hardware/software and latency, also no callback support.
[~till.rohrmann]
> flink as micro service
> ----------------------
>
> Key: FLINK-6085
> URL: https://issues.apache.org/jira/browse/FLINK-6085
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API, JobManager
> Reporter: Chen Qin
> Priority: Minor
> Attachments: Untitled document.jpg
>
>
> Track discussion around run flink as a micro service, includes but not
> limited to
> - RPC (web service endpoint) source
> as web service endpoint accept RPC call, ingest to the streaming job(only
> one)
> - callback mechanism
> - task assignment should honor deployment group (web tier hosts should be
> isolated from rest of task assignment)
> https://docs.google.com/document/d/1MSsTOz7xUu50dAf_8v3gsQFfJFFy9LKnULdIl26yj0o/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
