tianhui5 commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-678068937
> > > Thank you for you effort that you spend on this issue, here is my
suggestion, what do you think?
> > > If `offset` is null, please just set it to zero.
> >
> >
> > I wonder if there is any difference between setting offset to zero and
seekToBeginning.
>
> Please correct me if I was wrong.
> Let take the above test by me as an example, after scale up, it should
continue its consumptation from this position:
>
> ```json
> {
> "0":363171,
> "1":363198,
> "2":363249,
> "3":363171,
> "4":363199,
> "5":363250,
> "6":363170,
> "7":363196,
> "8":363250,
> "9":363170,
> "10":0,
> "11":0,
> "12":0,
> "13":0,
> "14":0,
> "15":0
> }
> ```
>
> not this position:
>
> ```json
> {
> "0":0,
> "1":0,
> "2":0,
> "3":0,
> "4":0,
> "5":0,
> "6":0,
> "7":0,
> "8":0,
> "9":0,
> "10":0,
> "11":0,
> "12":0,
> "13":0,
> "14":0,
> "15":0
> }
> ```
>
> So, if you are using `seekToBeginning` API, can you make sure the
partition 0 to partition 9 consume at right position?
Actually, I just add new partitions to list which seekToBeging. So the old
partitions' offset won't be effected. Here is my test when I scale kafka from 7
partitions to 10 partitions.
2020-08-21 14:18:02,634 INFO [qtp310350177-45] kafka.KafkaSource:192 :
localConsumeStats from local checkpoint null, remoteConsumeStats from remote
checkpoint {"0":1369,"1":1412,"2":1500,"3":1395,"4":1430,"5":0,"6":0}
2020-08-21 14:24:33,038 INFO [cost_test_channel]
storage.StreamingSegmentManager:645 : Print check point for cube cost_test
,CheckPoint{sourceConsumePosition='{"0":1408,"4":1476,"9":0}',
persistedIndexes={1597978800000=1, 1597986000000=1, 1597982400000=1,
1597975200000=1, 1597989600000=3},
longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={},
totalLongLatencyEventCnt=0},
segmentSourceStartPosition={1597978800000={"0":1376,"4":1442},
1597986000000={"0":1398,"4":1457}, 1597982400000={"0":1392,"4":1454},
1597975200000={"0":1369,"4":1430}, 1597989600000={"0":1408,"4":1467,"9":0}},
checkPointTime=1597991073038, totalCount=88, checkPointCount=1}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
