[
https://issues.apache.org/jira/browse/KYLIN-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yaqian Zhang closed KYLIN-4983.
-------------------------------
Resolved in release 3.1.3 (2022-01-06)
> The stream cube will be paused when user append a batch segment first
> ---------------------------------------------------------------------
>
> Key: KYLIN-4983
> URL: https://issues.apache.org/jira/browse/KYLIN-4983
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
> Reporter: Kun Liu
> Assignee: Kun Liu
> Priority: Blocker
> Fix For: v3.1.3
>
>
> Env:
> stream cube (stream_cube) with lambda, and the window of cube is 1hour
> Before enable the stream_cube, we submit a build restful request with the
> range [2025-01-01,2025-01-02]
>
> Then enable the stream_cube, the receiver node will consume data and create
> segment in real time, but the start time of the new segment is must be less
> than `2025-01-01`.
>
> When the receiver node upload the data of the segment to the HDFS and notify
> the coordinator, and the coordinator will delete the metadata of the stream
> segment like below code, but the segment of data can't be delete in the
> receiver node.
>
> ```
> // If we have a exist historical segment, we should not let new realtime
> segment overwrite it, it is so dangrous,
> // we just delete the entry to ignore the segment which should not exist
> if (segmentRange.getFirst() < minSegmentStart) {
> logger.warn(
> "The cube segment state is not correct because it belongs to historical
> part, cube:{} segment:{}, clear it.",
> cubeName, segmentState.getSegmentName());
> coordinator.getStreamMetadataStore().removeSegmentBuildState(cubeName,
> segmentState.getSegmentName());
> continue;
> }
> ```
> The number of immutable segment will reach to 100.
>
> There are two way to resolve this issue:
>
> # forbid appending segment in stream with lambda for the restful api
> # delete local data and remote hdfs data before remove metadata
> Now i think it is a better way to forbid appending segment in lambda stream
> cube.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
