??????Flink????DAG??????????????????????

Sat, 09 Nov 2019 06:10:59 -0800 

????????????????????????????????????????????????????source????????????????????????????????????????????????????????????????source??????????????????????


????????????????????????????
??



------------------ ???????? ------------------
??????: ???? <[email protected]&gt;
????????: 2019??11??9?? 21:59
??????: user-zh <[email protected]&gt;
????: user-zh <[email protected]&gt;
????: ??????Flink????DAG??????????????????????



4??5????????????????????????




| |
????
|
|
[email protected]
|

?????? ???????????? ????

??2019??11??09?? 21:50???????? ??????
????????????????????????


????????kafka????????????????????????????????????????????????





------------------ ???????? ------------------
??????: ???? <[email protected]&amp;gt;
????????: 2019??11??9?? 20:56
??????: user-zh <[email protected]&amp;gt;
????: ??????Flink????DAG??????????????????????



????????????????????case????????????????????????????????????????????
????1??????????source-&amp;gt;????????-&amp;gt;????????????tmp??
????2??????????tmp-&amp;gt;??????????????a-&amp;gt;??????
????3??????????tmp-&amp;gt;????????????b-&amp;gt;??????


??????????????????????????DAG????????????
1.val source = ds from KafkaConnector;
2.val broadcast = ruleMap.broacast();
3.val tmp = source.connect(broadcast).process();&amp;nbsp;
4.val a = tmp.map().addSink();&amp;nbsp;
5.val b = tmp.flatmap().keyBy( 8 fields ).window( 5min ).aggregate( sum 
).addSink();


??????????3????tmp????????????????????????????4??sink????????????????????????5??sink??????????????4??5????sink
 to 
mysql????????????????????????????5????????????job??????5??tps??????????200W/s??????case??????????????????????????????????????????source????????????????????????????????
??????????????????????????????????
??????????????????????????????5????????????????????????4??
??????????????????????????
1??????5????????????????????????flatmap??1????????????500??????????tps??:500*200W/s??keyBy??8????????aggr??sum??
2??????5??????????????job??????????1??????????????????????1??????????????????3??????????3????????????????????4??
3??????5??????????????sink????????mysql??mysql??????IO????????
??????????3????????????????????????????????5??sink??????kafka??????mq??????????mq??????mysql??


?????????? flink??????????????????????????????????????????????

回复