[
https://issues.apache.org/jira/browse/KYLIN-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17370169#comment-17370169
]
ASF GitHub Bot commented on KYLIN-4983:
---------------------------------------
hit-lacus merged pull request #1644:
URL: https://github.com/apache/kylin/pull/1644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> The stream cube will be paused when user append a batch segment first
> ---------------------------------------------------------------------
>
> Key: KYLIN-4983
> URL: https://issues.apache.org/jira/browse/KYLIN-4983
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
> Reporter: Kun Liu
> Assignee: Kun Liu
> Priority: Blocker
> Fix For: v3.1.3
>
>
> Env:
> stream cube (stream_cube) with lambda, and the window of cube is 1hour
> Before enable the stream_cube, we submit a build restful request with the
> range [2025-01-01,2025-01-02]
>
> Then enable the stream_cube, the receiver node will consume data and create
> segment in real time, but the start time of the new segment is must be less
> than `2025-01-01`.
>
> When the receiver node upload the data of the segment to the HDFS and notify
> the coordinator, and the coordinator will delete the metadata of the stream
> segment like below code, but the segment of data can't be delete in the
> receiver node.
>
> ```
> // If we have a exist historical segment, we should not let new realtime
> segment overwrite it, it is so dangrous,
> // we just delete the entry to ignore the segment which should not exist
> if (segmentRange.getFirst() < minSegmentStart) {
> logger.warn(
> "The cube segment state is not correct because it belongs to historical
> part, cube:{} segment:{}, clear it.",
> cubeName, segmentState.getSegmentName());
> coordinator.getStreamMetadataStore().removeSegmentBuildState(cubeName,
> segmentState.getSegmentName());
> continue;
> }
> ```
> The number of immutable segment will reach to 100.
>
> There are two way to resolve this issue:
>
> # forbid appending segment in stream with lambda for the restful api
> # delete local data and remote hdfs data before remove metadata
> Now i think it is a better way to forbid appending segment in lambda stream
> cube.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
