Hi Squirrels,

I have some trouble expressing my use case in Flink terms, so I am asking
for your help:
I have five million documents and fourteen million search terms. For each
search term I want to know, in how many documents it occurs. So basically a
`grep` with very many search terms.

My current approach is to broadcast the search terms to all nodes, then, at
each node, load them into a prefix tree/trie, which allows me to do the
contains check faster, and then process a share of the documents at each
node.
However, not all search terms fit into the main memory at each node (I
already set taskmanager.memory.fraction to 0.5), so I basically need to run
three batches with one third of the search terms each.

Questions:
(1): Is there a better way to express this in Flink? Distributing the nine
million search terms (around 300 MB) at a time to ten nodes does not seem
like the best idea. Also, the documentation says "As the content of
broadcast variables is kept in-memory on each node, it should not become
too large", however a simple join would be too big, and not allow me to use
a tree structure.
(2): Is there a way to execute the three batches inside one Flink program,
one after another? Currently, I need to run my program three times with a
different parameter, download from hdfs, concat, upload to hdfs ... to get
my final result. The three batches may not run in parallel, as I need the
available memory for the tree and get memory exceptions otherwise.
(3) [May be obsolete, if there is a better solution in (1)]: In the
broadcast scenario, I guess (?) I end up storing the search terms twice.
Once in the broadcast data set and once in my tree data structure. Is there
a way, to free the broadcast store on each node, after I copied it in my
data structure? This would free some more memory.

Thank you for any help,
Stefan

Reply via email to