Are you talking about a streaming or a batch job?
You are mentioning a "text stream" but also say you want to stream 100TB
-- indicating you have a finite data set using DataSet API.
-Matthias
On 05/22/2016 09:50 PM, Xtra Coder wrote:
> Hello,
>
> Question from newbie about how Flink's WordCount will actually work at
> scale.
>
> I've read/seen rather many high-level presentations and do not see
> more-or-less clear answers for following …
>
> Use-case:
> --
> there is huuuge text stream with very variable set of words – let's say
> 1BLN of unique words. Storing them just as raw text, without
> supplementary data, will take roughly 16TB of RAM. How Flink is
> approaching this internally.
>
> Here I'm more interested in following:
> 1. How individual words are spread in cluster of Flink nodes?
> Will each word appear exactly in one node and will be counted there or
> ... I'm not sure about the variants
>
> 2. As far as I understand – while job is running all its intermediate
> aggregation results are stored in-memory across cluster nodes (which may
> be partially written to local drive).
> Wild guess - what size of cluster is required to run above mentioned
> tasks efficiently?
>
> And two functional question on top of this ...
>
> 1. Since intermediate results are in memory – I guess it should be
> possible to get “current” counter for any word being processed.
> Is this possible?
>
> 2. After I've streamed 100TB of text – what will be the right way to
> save result to HDFS. For example I want to save list of words ordered by
> key with portions of 10mln per file compressed with bzip2.
> What APIs I should use?
> Since Flink uses intermediate snapshots for falt-tolerance - is it
> possible to save whole "current" state without stopping the stream?
>
> Thanks.
signature.asc
Description: OpenPGP digital signature
