?????? Re: flink????kafka????????????kafka??????????????????

Evan Sun, 12 Jan 2020 18:49:05 -0800

??????????kafka??????Offset??????????
kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper zkhost:2181 
--group ${group.id} --topic ${topic_name}
????zkhost ??group.id??topic_name????
????????????
Group&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Topic&nbsp; &nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Pid Offset&nbsp; 
&nbsp; &nbsp; &nbsp; &nbsp; logSize&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Lag&nbsp; 
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Owner
test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; dy_event&nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; 
&nbsp;8115733&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;10658588&nbsp; &nbsp; &nbsp; 
&nbsp; 2542855&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;none
test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; dy_event&nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1&nbsp; 
&nbsp;8114221&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;10658585&nbsp; &nbsp; &nbsp; 
&nbsp; 2544364&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;none
test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; dy_event&nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;2&nbsp; 
&nbsp;8115173&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;10658587&nbsp; &nbsp; &nbsp; 
&nbsp; 2543414&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;none
test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; dy_event&nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;3&nbsp; 
&nbsp;8115127&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;10658585&nbsp; &nbsp; &nbsp; 
&nbsp; 2543458&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;none
test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; dy_event&nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4&nbsp; 
&nbsp;8115160&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;10658587&nbsp; &nbsp; &nbsp; 
&nbsp; 2543427&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;none




????pid Offset??????????????????????????????????




------------------&nbsp;????????&nbsp;------------------
??????:&nbsp;"Benchao Li"<[email protected]&gt;;
????????:&nbsp;2020??1??12??(??????) ????9:46
??????:&nbsp;"user-zh"<[email protected]&gt;;

????:&nbsp;Re: Re: flink????kafka????????????kafka??????????????????



Flink????commit 
offset??Kafka??[1]??????????checkpoint??????checkpoint????????????????????

offset
??????commit????kafka??????offset????????????????????????commit??offset??????????????????????????????????????flink??????topic??????offset??????????check????????????????commit
offset.

[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/kafka.html#kafka-consumers-offset-committing-behaviour-configuration

wqpapa <[email protected]&gt; ??2020??1??12?????? ????9:34??????

&gt; 
????????????????????????????kafka??????????????????????????????????????????????????????java????????kafka??????????kafka-consumer-groups.sh
&gt; 
--describe????????????????????????????flink????????????????????kafka????????????????
&gt;
&gt;
&gt;
&gt;
&gt;
&gt;
&gt;
&gt;
&gt;
&gt; ?? 2020-01-12 21:17:40??"Benchao Li" <[email protected]&gt; ??????
&gt; &gt;setStartFromGroupOffsets
&gt;
&gt; 
&gt;????????kafka??????????offset??????????flink??????????kafka????????????consumer??offset??????????????????????kafka
&gt; &gt;consumer??????'auto.offset.reset'??????????partition??offset??
&gt; &gt;
&gt; &gt;????????????????????[1].
&gt; &gt;
&gt; &gt;[1]
&gt; &gt;
&gt; 
https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/kafka.html#kafka-consumers-start-position-configuration
&gt; &gt;
&gt; &gt;wqpapa <[email protected]&gt; ??2020??1??12?????? ????9:09??????
&gt; &gt;
&gt; &gt;&gt; flink????FlinkKafkaConsumer????kafka??????????group.id
&gt; &gt;&gt; 
??setStartFromGroupOffsets??????????????kafka??????????group.id????????????
&gt; &gt;&gt; 
kafka??0.9????????zk????????????????group.id??????kafka-consumer-groups.sh 
--describe
&gt; &gt;&gt; --group ????????????????????????????????
&gt; &gt;
&gt; &gt;
&gt; &gt;
&gt; &gt;--
&gt; &gt;
&gt; &gt;Benchao Li
&gt; &gt;School of Electronics Engineering and Computer Science, Peking 
University
&gt; &gt;Tel:+86-15650713730
&gt; &gt;Email: [email protected]; [email protected]
&gt;


-- 

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:+86-15650713730
Email: [email protected]; [email protected]

?????? Re: flink????kafka????????????kafka??????????????????

回复