Hi Averell, Did you know Flink allow you to customize a partitioner?
Some resource : official documentation : https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/operators/#physical-partitioning discussing in mailing list : http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/keyBy-using-custom-partitioner-td5379.html a code example : https://gist.github.com/chiwanpark/e71d27cc8edae8bc7298 Thanks, vino. 2018-07-30 18:57 GMT+08:00 Averell <lvhu...@gmail.com>: > Hi everyone, > > We are collecting log files from tens of thousands of network nodes, and we > need to do some data insights using that. The files are coming with the > corresponding node ID in the file name, and I want to do custom > partitioning > using that Node ID. > Right now (with Flink 1.5) I think that is not supported. I have been > trying > to look into the code, but it would take some time for me to understand. > From the GUI, it looks like the first step of file source (directory > monitoring) is rebalancing the stream to the 2nd step (file reader). And as > per Flink document, rebalancing means round-robin. However, I could not > find > the call of "rebalancing" method, but "transform" is called. Not much > information about that "transform" method though. > > Would it possible for me to ask for some guideline on this? > > Thanks for your help. > Averell > > > > -- > Sent from: http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/ >
